2020-11-19T02:32:25.237 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-19T02:32:25.239 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --block --lock --verbose --archive /home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633829 --owner scheduled_yuvalif@teuthology --name yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi -- /tmp/teuthology-worker.hmvh7_tg.tmp 2020-11-19T02:32:25.270 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-19T02:32:25.333 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633829 branch: wip-yuval-fix-47904 description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: null first_in_suite: false job_id: '5633829' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-yuval-fix-47904 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-yuval-fix-47904 sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 owner: scheduled_yuvalif@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuval-fix-47904 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.6672 2020-11-19T02:32:25.334 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa; will attempt to use it 2020-11-19T02:32:25.335 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa/tasks 2020-11-19T02:32:25.335 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-19T02:32:25.337 INFO:teuthology.task.internal:Checking packages... 2020-11-19T02:32:25.370 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '409eac507e130adcc9fee71aa5fd16ed7f289313' 2020-11-19T02:32:25.370 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-19T02:32:25.371 INFO:teuthology.packaging:ref: None 2020-11-19T02:32:25.371 INFO:teuthology.packaging:tag: None 2020-11-19T02:32:25.371 INFO:teuthology.packaging:branch: wip-yuval-fix-47904 2020-11-19T02:32:25.372 INFO:teuthology.packaging:sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T02:32:25.372 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-yuval-fix-47904 2020-11-19T02:32:25.561 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7199-g409eac50-1bionic 2020-11-19T02:32:25.562 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-19T02:32:25.571 INFO:teuthology.task.internal:no buildpackages task found 2020-11-19T02:32:25.572 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-19T02:32:25.593 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-19T02:32:25.594 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-19T02:32:25.648 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:32:35.669 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:32:45.689 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:32:55.711 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:33:05.730 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:33:15.769 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:33:25.791 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:33:35.815 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:33:45.832 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:33:55.929 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:34:05.957 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:34:15.991 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:34:26.008 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:34:36.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:34:46.043 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:34:56.054 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:35:06.079 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:35:16.098 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:35:26.116 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:35:36.137 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:35:46.157 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T02:35:56.173 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T02:36:06.189 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T02:36:16.207 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T02:36:26.228 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T02:36:36.254 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T02:36:46.273 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T02:36:56.298 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T02:37:06.327 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T02:37:16.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T02:37:26.362 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_yuvalif@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633829'} 2020-11-19T02:37:26.384 DEBUG:teuthology.lock.ops:locked ubuntu@smithi086.front.sepia.ceph.com, ubuntu@smithi160.front.sepia.ceph.com 2020-11-19T02:37:26.443 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3b6697bc80>, signals=[15]) 2020-11-19T02:37:26.649 INFO:teuthology.provision.fog.smithi086:Scheduling deploy of ubuntu 18.04 2020-11-19T02:37:26.652 INFO:teuthology.provision.fog.smithi160:Scheduling deploy of ubuntu 18.04 2020-11-19T02:37:26.994 INFO:teuthology.orchestra.console:Power off smithi160 2020-11-19T02:37:26.996 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-19T02:37:27.024 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T02:37:27.066 INFO:teuthology.orchestra.console:Power off smithi086 2020-11-19T02:37:27.067 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-19T02:37:27.087 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T02:37:27.147 INFO:teuthology.orchestra.console:Power off for smithi160 completed 2020-11-19T02:37:27.212 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2020-11-19T02:37:27.249 INFO:teuthology.orchestra.console:Power on smithi160 2020-11-19T02:37:27.249 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-19T02:37:27.313 INFO:teuthology.orchestra.console:Power on smithi086 2020-11-19T02:37:27.314 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-19T02:37:27.363 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T02:37:27.395 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T02:37:31.395 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T02:37:31.465 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T02:37:35.512 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T02:37:35.575 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T02:37:35.723 INFO:teuthology.orchestra.console:Power on for smithi160 completed 2020-11-19T02:37:35.785 INFO:teuthology.orchestra.console:Power on for smithi086 completed 2020-11-19T02:37:35.826 INFO:teuthology.provision.fog.smithi160:Waiting for deploy to finish 2020-11-19T02:37:35.887 INFO:teuthology.provision.fog.smithi086:Waiting for deploy to finish 2020-11-19T02:40:37.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:40:37.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:01.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:01.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:10.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:10.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:19.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:19.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:28.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:28.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:37.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:41:46.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:42:34.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:42:36.020 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-19T02:42:36.022 INFO:teuthology.orchestra.run.smithi086:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-19T02:42:36.403 INFO:teuthology.provision.fog.smithi086:Node is ready 2020-11-19T02:42:36.404 INFO:teuthology.orchestra.run.smithi086:> hostname 2020-11-19T02:42:36.452 INFO:teuthology.orchestra.run.smithi086.stdout:smithi086.front.sepia.ceph.com 2020-11-19T02:42:36.452 INFO:teuthology.orchestra.run.smithi086:> grep smithi086.front.sepia.ceph.com /etc/hosts 2020-11-19T02:42:36.500 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.15.86 smithi086.front.sepia.ceph.com smithi086 2020-11-19T02:42:36.501 INFO:teuthology.orchestra.run.smithi086:> sudo hostname smithi086 2020-11-19T02:42:36.595 INFO:teuthology.orchestra.run.smithi086:> sudo sed -i -e 's/smithi086.front.sepia.ceph.com/smithi086/g' /etc/hosts 2020-11-19T02:42:36.613 INFO:teuthology.orchestra.run.smithi086:> sudo sed -i -e 's/172.21.15.86/172.21.15.86/g' /etc/hosts 2020-11-19T02:42:36.668 INFO:teuthology.orchestra.run.smithi086:> sudo sed -i -e 's/smithi086.front.sepia.ceph.com/smithi086/g' /etc/hostname 2020-11-19T02:42:36.729 INFO:teuthology.orchestra.run.smithi086:> sudo hostname smithi086 2020-11-19T02:42:36.783 INFO:teuthology.provision.fog.smithi086:Deploy complete! 2020-11-19T02:42:36.784 DEBUG:teuthology.parallel:result is None 2020-11-19T02:42:52.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:42:53.097 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-19T02:42:53.098 INFO:teuthology.orchestra.run.smithi160:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-19T02:42:53.439 INFO:teuthology.provision.fog.smithi160:Node is ready 2020-11-19T02:42:53.439 INFO:teuthology.orchestra.run.smithi160:> hostname 2020-11-19T02:42:53.485 INFO:teuthology.orchestra.run.smithi160.stdout:smithi160.front.sepia.ceph.com 2020-11-19T02:42:53.486 INFO:teuthology.orchestra.run.smithi160:> grep smithi160.front.sepia.ceph.com /etc/hosts 2020-11-19T02:42:53.534 INFO:teuthology.orchestra.run.smithi160.stdout:172.21.15.160 smithi160.front.sepia.ceph.com smithi160 2020-11-19T02:42:53.535 INFO:teuthology.orchestra.run.smithi160:> sudo hostname smithi160 2020-11-19T02:42:53.636 INFO:teuthology.orchestra.run.smithi160:> sudo sed -i -e 's/smithi160.front.sepia.ceph.com/smithi160/g' /etc/hosts 2020-11-19T02:42:53.650 INFO:teuthology.orchestra.run.smithi160:> sudo sed -i -e 's/172.21.15.160/172.21.15.160/g' /etc/hosts 2020-11-19T02:42:53.707 INFO:teuthology.orchestra.run.smithi160:> sudo sed -i -e 's/smithi160.front.sepia.ceph.com/smithi160/g' /etc/hostname 2020-11-19T02:42:53.762 INFO:teuthology.orchestra.run.smithi160:> sudo hostname smithi160 2020-11-19T02:42:53.818 INFO:teuthology.provision.fog.smithi160:Deploy complete! 2020-11-19T02:42:53.819 DEBUG:teuthology.parallel:result is None 2020-11-19T02:42:53.956 INFO:teuthology.lock.ops:Checking smithi086.front.sepia.ceph.com 2020-11-19T02:42:53.957 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-19T02:42:53.974 INFO:teuthology.lock.ops:Checking smithi160.front.sepia.ceph.com 2020-11-19T02:42:53.975 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-19T02:42:53.991 INFO:teuthology.lock.ops:Updating [smithi086.front.sepia.ceph.com]: set os type and version on server 2020-11-19T02:42:53.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T02:42:53.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:42:54.066 INFO:teuthology.orchestra.run.smithi086:> true 2020-11-19T02:42:54.307 INFO:teuthology.orchestra.run.smithi086:> uname -m 2020-11-19T02:42:54.355 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2020-11-19T02:42:54.355 INFO:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2020-11-19T02:42:54.403 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Ubuntu" 2020-11-19T02:42:54.403 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-19T02:42:54.403 INFO:teuthology.orchestra.run.smithi086.stdout:ID=ubuntu 2020-11-19T02:42:54.403 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE=debian 2020-11-19T02:42:54.404 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-19T02:42:54.404 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="18.04" 2020-11-19T02:42:54.404 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-19T02:42:54.404 INFO:teuthology.orchestra.run.smithi086.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-19T02:42:54.404 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-19T02:42:54.405 INFO:teuthology.orchestra.run.smithi086.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-19T02:42:54.405 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_CODENAME=bionic 2020-11-19T02:42:54.405 INFO:teuthology.orchestra.run.smithi086.stdout:UBUNTU_CODENAME=bionic 2020-11-19T02:42:54.406 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2020-11-19T02:42:54.421 INFO:teuthology.lock.ops:Updating [smithi160.front.sepia.ceph.com]: set os type and version on server 2020-11-19T02:42:54.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T02:42:54.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:42:54.497 INFO:teuthology.orchestra.run.smithi160:> true 2020-11-19T02:42:54.695 INFO:teuthology.orchestra.run.smithi160:> uname -m 2020-11-19T02:42:54.742 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2020-11-19T02:42:54.743 INFO:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2020-11-19T02:42:54.790 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="Ubuntu" 2020-11-19T02:42:54.790 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-19T02:42:54.790 INFO:teuthology.orchestra.run.smithi160.stdout:ID=ubuntu 2020-11-19T02:42:54.791 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE=debian 2020-11-19T02:42:54.791 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-19T02:42:54.791 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="18.04" 2020-11-19T02:42:54.791 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-19T02:42:54.791 INFO:teuthology.orchestra.run.smithi160.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-19T02:42:54.792 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-19T02:42:54.792 INFO:teuthology.orchestra.run.smithi160.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-19T02:42:54.792 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_CODENAME=bionic 2020-11-19T02:42:54.792 INFO:teuthology.orchestra.run.smithi160.stdout:UBUNTU_CODENAME=bionic 2020-11-19T02:42:54.793 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2020-11-19T02:42:54.811 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-19T02:42:54.844 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi086.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOX2yzewpn2i4e07+3TrL99b0P2l1x7R1S9+FHleR/gamdR1+xV0/Rm+YdgPbTxuw+O/rQODT7fLlVl6UBihJS2pfoS+X9X0o1znO23OYTzkehepKz6xqNXkHKnxEdAvddqlisgyrYnC5j6KNiWIlSybv25VJ//uZQqrkKU74Uz2fp6l83hoQjttdrZoowG1KmXEz5gdqczArtjOO7oL81jf7RAY5Abrs0XdnVq0ifxD6lSpsxlv3HiEO5/E4FhEErv5xcawzDG8Nle9rjLjnO9QNp6H87gLyfTj72zE0fgSHbB5F08vO/AxZhZynig/FWjffG2ftCmO9tOywwHVz9 smithi160.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDwLcfm2rpI0BEMcouJXOuPRNnfF7ta3SKliynVLHNZbLhzUxe85D2ZtcpDnU2EwXhCEv37c+wXgbEu92FsekOyP246U5k4aX74T8LiGLHuvCXgyvqwq9qjWsDFS4oYn6yuCUzv9aVMpkX6K8eq7vmknRVFEA3Xf3hWWIUeltVOeapmEutN6AlNm+dRfX8rNqiYubr8cWyr5nHsDx6uhjTJBnu1HDghlbttSZNPzeOy95IRTqQUXJHk0G+zBjG5wf+FaeeReGIj/ttMqUQsyHgT5U+zjyGJm0fCTL7ZQF4Rh0e0x+Bj4cq1FLcvNUSXUHmH8q8o0B40fXWovLgRPNbh 2020-11-19T02:42:54.845 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-19T02:42:54.895 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-19T02:42:54.903 INFO:teuthology.task.internal:Saving configuration 2020-11-19T02:42:54.921 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-19T02:42:54.932 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-19T02:42:54.949 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-19 02:37:26.370062', 'description': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633829', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII53fADizh3UsWFRQz7Jcj8UTn2Gev2TJQTVja1KZOLL', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi086.front.sepia.ceph.com'} 2020-11-19T02:42:54.965 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-19 02:37:26.370081', 'description': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633829', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHiyqADGEb651m9RrUNG/4eXLHlDLbZltOyfzhC/T5l4', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi160.front.sepia.ceph.com'} 2020-11-19T02:42:54.966 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-19T02:42:54.977 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-11-19T02:42:54.977 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-11-19T02:42:54.978 INFO:teuthology.run_tasks:Running task console_log... 2020-11-19T02:42:55.041 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3b6a4a6e18>, signals=[15]) 2020-11-19T02:42:55.042 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-19T02:42:55.052 INFO:teuthology.task.internal:Opening connections... 2020-11-19T02:42:55.053 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2020-11-19T02:42:55.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:42:55.133 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2020-11-19T02:42:55.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:42:55.210 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-19T02:42:55.221 INFO:teuthology.orchestra.run.smithi086:> uname -m 2020-11-19T02:42:55.333 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2020-11-19T02:42:55.334 INFO:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2020-11-19T02:42:55.381 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Ubuntu" 2020-11-19T02:42:55.381 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-19T02:42:55.381 INFO:teuthology.orchestra.run.smithi086.stdout:ID=ubuntu 2020-11-19T02:42:55.382 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE=debian 2020-11-19T02:42:55.382 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-19T02:42:55.382 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="18.04" 2020-11-19T02:42:55.382 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-19T02:42:55.382 INFO:teuthology.orchestra.run.smithi086.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-19T02:42:55.383 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-19T02:42:55.383 INFO:teuthology.orchestra.run.smithi086.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-19T02:42:55.383 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_CODENAME=bionic 2020-11-19T02:42:55.383 INFO:teuthology.orchestra.run.smithi086.stdout:UBUNTU_CODENAME=bionic 2020-11-19T02:42:55.384 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2020-11-19T02:42:55.404 INFO:teuthology.orchestra.run.smithi160:> uname -m 2020-11-19T02:42:55.410 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2020-11-19T02:42:55.411 INFO:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2020-11-19T02:42:55.458 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="Ubuntu" 2020-11-19T02:42:55.459 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-19T02:42:55.459 INFO:teuthology.orchestra.run.smithi160.stdout:ID=ubuntu 2020-11-19T02:42:55.459 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE=debian 2020-11-19T02:42:55.459 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-19T02:42:55.460 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="18.04" 2020-11-19T02:42:55.460 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-19T02:42:55.460 INFO:teuthology.orchestra.run.smithi160.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-19T02:42:55.460 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-19T02:42:55.460 INFO:teuthology.orchestra.run.smithi160.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-19T02:42:55.461 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_CODENAME=bionic 2020-11-19T02:42:55.461 INFO:teuthology.orchestra.run.smithi160.stdout:UBUNTU_CODENAME=bionic 2020-11-19T02:42:55.461 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2020-11-19T02:42:55.477 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-19T02:42:55.513 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-19T02:42:55.524 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-19T02:42:55.524 INFO:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2020-11-19T02:42:55.527 INFO:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2020-11-19T02:42:55.532 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-19T02:42:55.542 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-19T02:42:55.543 INFO:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2020-11-19T02:42:55.575 INFO:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2020-11-19T02:42:55.585 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-19T02:42:55.637 INFO:teuthology.run_tasks:Running task kernel... 2020-11-19T02:42:55.662 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-11-19T02:42:55.663 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-19T02:42:55.664 INFO:teuthology.orchestra.run.smithi086:> uname -r 2020-11-19T02:42:55.671 INFO:teuthology.orchestra.run.smithi086.stdout:4.15.0-123-generic 2020-11-19T02:42:55.671 INFO:teuthology.task.kernel:Running kernel on smithi086: 4.15.0-123-generic 2020-11-19T02:42:55.672 INFO:teuthology.orchestra.run.smithi086:> sudo apt-get clean 2020-11-19T02:42:55.782 INFO:teuthology.orchestra.run.smithi086:> sudo apt-get update 2020-11-19T02:42:56.050 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T02:42:56.051 INFO:teuthology.orchestra.run.smithi086.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-19T02:42:56.108 INFO:teuthology.orchestra.run.smithi086.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-19T02:42:56.380 INFO:teuthology.orchestra.run.smithi086.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-19T02:42:56.456 INFO:teuthology.orchestra.run.smithi086.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [861 kB] 2020-11-19T02:42:56.591 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,162 kB] 2020-11-19T02:42:56.813 INFO:teuthology.orchestra.run.smithi086.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,426 kB] 2020-11-19T02:42:56.919 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,088 kB] 2020-11-19T02:42:56.954 INFO:teuthology.orchestra.run.smithi086.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,761 kB] 2020-11-19T02:42:56.965 INFO:teuthology.orchestra.run.smithi086.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [966 kB] 2020-11-19T02:42:57.056 INFO:teuthology.orchestra.run.smithi086.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [373 kB] 2020-11-19T02:42:57.073 INFO:teuthology.orchestra.run.smithi086.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [196 kB] 2020-11-19T02:42:57.081 INFO:teuthology.orchestra.run.smithi086.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [17.3 kB] 2020-11-19T02:42:57.082 INFO:teuthology.orchestra.run.smithi086.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [26.3 kB] 2020-11-19T02:42:57.083 INFO:teuthology.orchestra.run.smithi086.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,692 kB] 2020-11-19T02:42:57.129 INFO:teuthology.orchestra.run.smithi086.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,547 kB] 2020-11-19T02:42:57.166 INFO:teuthology.orchestra.run.smithi086.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [355 kB] 2020-11-19T02:42:57.190 INFO:teuthology.orchestra.run.smithi086.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [36.1 kB] 2020-11-19T02:42:57.191 INFO:teuthology.orchestra.run.smithi086.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.4 kB] 2020-11-19T02:42:57.191 INFO:teuthology.orchestra.run.smithi086.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,308 B] 2020-11-19T02:42:58.325 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 11.8 MB in 2s (5,079 kB/s) 2020-11-19T02:42:59.716 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2020-11-19T02:42:59.763 INFO:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-19T02:42:59.839 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2020-11-19T02:43:00.090 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2020-11-19T02:43:00.090 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2020-11-19T02:43:00.169 INFO:teuthology.orchestra.run.smithi086.stderr:E: Unable to locate package linux-image-current-generic 2020-11-19T02:43:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-19T02:43:00.171 INFO:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-19T02:43:00.253 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2020-11-19T02:43:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2020-11-19T02:43:00.500 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2020-11-19T02:43:00.672 INFO:teuthology.orchestra.run.smithi086.stdout:The following additional packages will be installed: 2020-11-19T02:43:00.672 INFO:teuthology.orchestra.run.smithi086.stdout: linux-generic linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-19T02:43:00.672 INFO:teuthology.orchestra.run.smithi086.stdout: linux-headers-generic linux-image-4.15.0-124-generic 2020-11-19T02:43:00.673 INFO:teuthology.orchestra.run.smithi086.stdout: linux-modules-4.15.0-124-generic linux-modules-extra-4.15.0-124-generic 2020-11-19T02:43:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:Suggested packages: 2020-11-19T02:43:00.674 INFO:teuthology.orchestra.run.smithi086.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-11-19T02:43:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:Recommended packages: 2020-11-19T02:43:00.674 INFO:teuthology.orchestra.run.smithi086.stdout: thermald 2020-11-19T02:43:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:The following NEW packages will be installed: 2020-11-19T02:43:00.699 INFO:teuthology.orchestra.run.smithi086.stdout: linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-19T02:43:00.699 INFO:teuthology.orchestra.run.smithi086.stdout: linux-image-4.15.0-124-generic linux-modules-4.15.0-124-generic 2020-11-19T02:43:00.700 INFO:teuthology.orchestra.run.smithi086.stdout: linux-modules-extra-4.15.0-124-generic 2020-11-19T02:43:00.700 INFO:teuthology.orchestra.run.smithi086.stdout:The following packages will be upgraded: 2020-11-19T02:43:00.701 INFO:teuthology.orchestra.run.smithi086.stdout: linux-generic linux-headers-generic linux-image-generic 2020-11-19T02:43:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:3 upgraded, 5 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T02:43:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:Need to get 65.9 MB of archives. 2020-11-19T02:43:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:After this operation, 333 MB of additional disk space will be used. 2020-11-19T02:43:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-124-generic amd64 4.15.0-124.127 [13.1 MB] 2020-11-19T02:43:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-124-generic amd64 4.15.0-124.127 [8,012 kB] 2020-11-19T02:43:02.101 INFO:teuthology.orchestra.run.smithi086.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-124-generic amd64 4.15.0-124.127 [32.8 MB] 2020-11-19T02:43:03.033 INFO:teuthology.orchestra.run.smithi086.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.124.111 [1,868 B] 2020-11-19T02:43:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.124.111 [2,624 B] 2020-11-19T02:43:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124 all 4.15.0-124.127 [10.9 MB] 2020-11-19T02:43:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124-generic amd64 4.15.0-124.127 [1,084 kB] 2020-11-19T02:43:03.356 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.124.111 [2,520 B] 2020-11-19T02:43:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 65.9 MB in 3s (25.0 MB/s) 2020-11-19T02:43:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-modules-4.15.0-124-generic. 2020-11-19T02:43:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-11-19T02:43:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../0-linux-modules-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T02:43:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:05.664 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-image-4.15.0-124-generic. 2020-11-19T02:43:05.677 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../1-linux-image-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T02:43:05.775 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:06.284 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-124-generic. 2020-11-19T02:43:06.296 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T02:43:06.334 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:09.967 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../3-linux-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T02:43:10.752 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T02:43:10.752 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T02:43:10.752 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-image-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T02:43:10.824 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-headers-4.15.0-124. 2020-11-19T02:43:10.836 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../5-linux-headers-4.15.0-124_4.15.0-124.127_all.deb ... 2020-11-19T02:43:10.874 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-19T02:43:15.391 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-headers-4.15.0-124-generic. 2020-11-19T02:43:15.413 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../6-linux-headers-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T02:43:15.459 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:17.791 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T02:43:17.912 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-headers-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T02:43:18.234 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:18.842 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:19.278 INFO:teuthology.orchestra.run.smithi086.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-124-generic 2020-11-19T02:43:19.278 INFO:teuthology.orchestra.run.smithi086.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-124-generic 2020-11-19T02:43:19.419 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:19.897 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-19T02:43:20.021 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-image-generic (4.15.0.124.111) ... 2020-11-19T02:43:20.130 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:20.231 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-headers-generic (4.15.0.124.111) ... 2020-11-19T02:43:20.357 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-generic (4.15.0.124.111) ... 2020-11-19T02:43:20.484 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:20.689 INFO:teuthology.orchestra.run.smithi086.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-11-19T02:43:20.689 INFO:teuthology.orchestra.run.smithi086.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-124-generic 2020-11-19T02:43:35.052 INFO:teuthology.orchestra.run.smithi086.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-11-19T02:43:35.053 INFO:teuthology.orchestra.run.smithi086.stdout:Sourcing file `/etc/default/grub' 2020-11-19T02:43:35.082 INFO:teuthology.orchestra.run.smithi086.stdout:Generating grub configuration file ... 2020-11-19T02:43:35.492 INFO:teuthology.orchestra.run.smithi086.stdout:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-19T02:43:35.521 INFO:teuthology.orchestra.run.smithi086.stdout:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-19T02:43:35.882 INFO:teuthology.orchestra.run.smithi086.stdout:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-19T02:43:35.910 INFO:teuthology.orchestra.run.smithi086.stdout:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-19T02:43:36.796 INFO:teuthology.orchestra.run.smithi086.stdout:done 2020-11-19T02:43:37.202 INFO:teuthology.orchestra.run.smithi086:> dpkg -s linux-image-generic 2020-11-19T02:43:37.226 INFO:teuthology.orchestra.run.smithi086.stdout:Package: linux-image-generic 2020-11-19T02:43:37.227 INFO:teuthology.orchestra.run.smithi086.stdout:Status: install ok installed 2020-11-19T02:43:37.227 INFO:teuthology.orchestra.run.smithi086.stdout:Priority: optional 2020-11-19T02:43:37.227 INFO:teuthology.orchestra.run.smithi086.stdout:Section: kernel 2020-11-19T02:43:37.227 INFO:teuthology.orchestra.run.smithi086.stdout:Installed-Size: 16 2020-11-19T02:43:37.228 INFO:teuthology.orchestra.run.smithi086.stdout:Maintainer: Ubuntu Kernel Team 2020-11-19T02:43:37.228 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: amd64 2020-11-19T02:43:37.228 INFO:teuthology.orchestra.run.smithi086.stdout:Source: linux-meta 2020-11-19T02:43:37.228 INFO:teuthology.orchestra.run.smithi086.stdout:Version: 4.15.0.124.111 2020-11-19T02:43:37.229 INFO:teuthology.orchestra.run.smithi086.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-19T02:43:37.229 INFO:teuthology.orchestra.run.smithi086.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-19T02:43:37.229 INFO:teuthology.orchestra.run.smithi086.stdout:Recommends: thermald 2020-11-19T02:43:37.229 INFO:teuthology.orchestra.run.smithi086.stdout:Description: Generic Linux kernel image 2020-11-19T02:43:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: This package will always depend on the latest generic kernel image 2020-11-19T02:43:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: available. 2020-11-19T02:43:37.230 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-123-generic Expected: 4.15.0-124-generic 2020-11-19T02:43:37.231 INFO:teuthology.orchestra.run.smithi160:> uname -r 2020-11-19T02:43:37.237 INFO:teuthology.orchestra.run.smithi160.stdout:4.15.0-123-generic 2020-11-19T02:43:37.237 INFO:teuthology.task.kernel:Running kernel on smithi160: 4.15.0-123-generic 2020-11-19T02:43:37.238 INFO:teuthology.orchestra.run.smithi160:> sudo apt-get clean 2020-11-19T02:43:37.350 INFO:teuthology.orchestra.run.smithi160:> sudo apt-get update 2020-11-19T02:43:37.611 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T02:43:37.612 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-19T02:43:37.675 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-19T02:43:37.954 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-19T02:43:39.063 INFO:teuthology.orchestra.run.smithi160.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [861 kB] 2020-11-19T02:43:39.064 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,761 kB] 2020-11-19T02:43:39.064 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,426 kB] 2020-11-19T02:43:39.064 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,088 kB] 2020-11-19T02:43:39.065 INFO:teuthology.orchestra.run.smithi160.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [966 kB] 2020-11-19T02:43:39.065 INFO:teuthology.orchestra.run.smithi160.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,162 kB] 2020-11-19T02:43:39.065 INFO:teuthology.orchestra.run.smithi160.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [373 kB] 2020-11-19T02:43:39.065 INFO:teuthology.orchestra.run.smithi160.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [17.3 kB] 2020-11-19T02:43:39.066 INFO:teuthology.orchestra.run.smithi160.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [196 kB] 2020-11-19T02:43:39.066 INFO:teuthology.orchestra.run.smithi160.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [26.3 kB] 2020-11-19T02:43:39.066 INFO:teuthology.orchestra.run.smithi160.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,692 kB] 2020-11-19T02:43:39.066 INFO:teuthology.orchestra.run.smithi160.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,547 kB] 2020-11-19T02:43:39.067 INFO:teuthology.orchestra.run.smithi160.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [355 kB] 2020-11-19T02:43:39.067 INFO:teuthology.orchestra.run.smithi160.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.4 kB] 2020-11-19T02:43:39.067 INFO:teuthology.orchestra.run.smithi160.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [36.1 kB] 2020-11-19T02:43:39.067 INFO:teuthology.orchestra.run.smithi160.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,308 B] 2020-11-19T02:43:40.201 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 11.8 MB in 2s (5,030 kB/s) 2020-11-19T02:43:41.648 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2020-11-19T02:43:41.697 INFO:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-19T02:43:41.778 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2020-11-19T02:43:42.035 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2020-11-19T02:43:42.035 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2020-11-19T02:43:42.119 INFO:teuthology.orchestra.run.smithi160.stderr:E: Unable to locate package linux-image-current-generic 2020-11-19T02:43:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-19T02:43:42.121 INFO:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-19T02:43:42.201 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2020-11-19T02:43:42.458 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2020-11-19T02:43:42.459 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2020-11-19T02:43:42.631 INFO:teuthology.orchestra.run.smithi160.stdout:The following additional packages will be installed: 2020-11-19T02:43:42.631 INFO:teuthology.orchestra.run.smithi160.stdout: linux-generic linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-19T02:43:42.631 INFO:teuthology.orchestra.run.smithi160.stdout: linux-headers-generic linux-image-4.15.0-124-generic 2020-11-19T02:43:42.632 INFO:teuthology.orchestra.run.smithi160.stdout: linux-modules-4.15.0-124-generic linux-modules-extra-4.15.0-124-generic 2020-11-19T02:43:42.633 INFO:teuthology.orchestra.run.smithi160.stdout:Suggested packages: 2020-11-19T02:43:42.633 INFO:teuthology.orchestra.run.smithi160.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-11-19T02:43:42.633 INFO:teuthology.orchestra.run.smithi160.stdout:Recommended packages: 2020-11-19T02:43:42.633 INFO:teuthology.orchestra.run.smithi160.stdout: thermald 2020-11-19T02:43:42.662 INFO:teuthology.orchestra.run.smithi160.stdout:The following NEW packages will be installed: 2020-11-19T02:43:42.662 INFO:teuthology.orchestra.run.smithi160.stdout: linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-19T02:43:42.662 INFO:teuthology.orchestra.run.smithi160.stdout: linux-image-4.15.0-124-generic linux-modules-4.15.0-124-generic 2020-11-19T02:43:42.662 INFO:teuthology.orchestra.run.smithi160.stdout: linux-modules-extra-4.15.0-124-generic 2020-11-19T02:43:42.663 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages will be upgraded: 2020-11-19T02:43:42.663 INFO:teuthology.orchestra.run.smithi160.stdout: linux-generic linux-headers-generic linux-image-generic 2020-11-19T02:43:42.837 INFO:teuthology.orchestra.run.smithi160.stdout:3 upgraded, 5 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T02:43:42.837 INFO:teuthology.orchestra.run.smithi160.stdout:Need to get 65.9 MB of archives. 2020-11-19T02:43:42.838 INFO:teuthology.orchestra.run.smithi160.stdout:After this operation, 333 MB of additional disk space will be used. 2020-11-19T02:43:42.838 INFO:teuthology.orchestra.run.smithi160.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-124-generic amd64 4.15.0-124.127 [13.1 MB] 2020-11-19T02:43:43.833 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-124-generic amd64 4.15.0-124.127 [8,012 kB] 2020-11-19T02:43:44.079 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-124-generic amd64 4.15.0-124.127 [32.8 MB] 2020-11-19T02:43:45.053 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.124.111 [1,868 B] 2020-11-19T02:43:45.054 INFO:teuthology.orchestra.run.smithi160.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.124.111 [2,624 B] 2020-11-19T02:43:45.054 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124 all 4.15.0-124.127 [10.9 MB] 2020-11-19T02:43:45.339 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124-generic amd64 4.15.0-124.127 [1,084 kB] 2020-11-19T02:43:45.379 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.124.111 [2,520 B] 2020-11-19T02:43:45.856 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 65.9 MB in 3s (24.4 MB/s) 2020-11-19T02:43:46.015 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-modules-4.15.0-124-generic. 2020-11-19T02:43:46.053 INFO:teuthology.orchestra.run.smithi160.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-11-19T02:43:46.056 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../0-linux-modules-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T02:43:46.091 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:47.724 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-image-4.15.0-124-generic. 2020-11-19T02:43:47.736 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../1-linux-image-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T02:43:47.837 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:48.352 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-124-generic. 2020-11-19T02:43:48.363 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T02:43:48.402 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:52.055 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../3-linux-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T02:43:52.176 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T02:43:52.509 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T02:43:52.638 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-image-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T02:43:52.967 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-headers-4.15.0-124. 2020-11-19T02:43:52.979 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../5-linux-headers-4.15.0-124_4.15.0-124.127_all.deb ... 2020-11-19T02:43:53.007 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-19T02:43:57.571 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-headers-4.15.0-124-generic. 2020-11-19T02:43:57.593 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../6-linux-headers-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T02:43:57.630 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:43:59.938 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T02:44:00.431 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-headers-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T02:44:00.432 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:44:00.956 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:44:01.394 INFO:teuthology.orchestra.run.smithi160.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-124-generic 2020-11-19T02:44:01.395 INFO:teuthology.orchestra.run.smithi160.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-124-generic 2020-11-19T02:44:01.523 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:44:01.991 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-19T02:44:02.126 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-image-generic (4.15.0.124.111) ... 2020-11-19T02:44:02.251 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:44:02.378 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-headers-generic (4.15.0.124.111) ... 2020-11-19T02:44:02.503 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-generic (4.15.0.124.111) ... 2020-11-19T02:44:02.638 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T02:44:02.839 INFO:teuthology.orchestra.run.smithi160.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-11-19T02:44:02.839 INFO:teuthology.orchestra.run.smithi160.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-124-generic 2020-11-19T02:44:17.593 INFO:teuthology.orchestra.run.smithi160.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-11-19T02:44:17.593 INFO:teuthology.orchestra.run.smithi160.stdout:Sourcing file `/etc/default/grub' 2020-11-19T02:44:17.623 INFO:teuthology.orchestra.run.smithi160.stdout:Generating grub configuration file ... 2020-11-19T02:44:17.992 INFO:teuthology.orchestra.run.smithi160.stdout:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-19T02:44:18.022 INFO:teuthology.orchestra.run.smithi160.stdout:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-19T02:44:18.348 INFO:teuthology.orchestra.run.smithi160.stdout:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-19T02:44:18.378 INFO:teuthology.orchestra.run.smithi160.stdout:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-19T02:44:19.678 INFO:teuthology.orchestra.run.smithi160.stdout:done 2020-11-19T02:44:20.094 INFO:teuthology.orchestra.run.smithi160:> dpkg -s linux-image-generic 2020-11-19T02:44:20.124 INFO:teuthology.orchestra.run.smithi160.stdout:Package: linux-image-generic 2020-11-19T02:44:20.125 INFO:teuthology.orchestra.run.smithi160.stdout:Status: install ok installed 2020-11-19T02:44:20.125 INFO:teuthology.orchestra.run.smithi160.stdout:Priority: optional 2020-11-19T02:44:20.125 INFO:teuthology.orchestra.run.smithi160.stdout:Section: kernel 2020-11-19T02:44:20.125 INFO:teuthology.orchestra.run.smithi160.stdout:Installed-Size: 16 2020-11-19T02:44:20.126 INFO:teuthology.orchestra.run.smithi160.stdout:Maintainer: Ubuntu Kernel Team 2020-11-19T02:44:20.126 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: amd64 2020-11-19T02:44:20.126 INFO:teuthology.orchestra.run.smithi160.stdout:Source: linux-meta 2020-11-19T02:44:20.127 INFO:teuthology.orchestra.run.smithi160.stdout:Version: 4.15.0.124.111 2020-11-19T02:44:20.127 INFO:teuthology.orchestra.run.smithi160.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-19T02:44:20.128 INFO:teuthology.orchestra.run.smithi160.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-19T02:44:20.128 INFO:teuthology.orchestra.run.smithi160.stdout:Recommends: thermald 2020-11-19T02:44:20.128 INFO:teuthology.orchestra.run.smithi160.stdout:Description: Generic Linux kernel image 2020-11-19T02:44:20.128 INFO:teuthology.orchestra.run.smithi160.stdout: This package will always depend on the latest generic kernel image 2020-11-19T02:44:20.129 INFO:teuthology.orchestra.run.smithi160.stdout: available. 2020-11-19T02:44:20.129 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-123-generic Expected: 4.15.0-124-generic 2020-11-19T02:44:20.130 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-11-19T02:44:20.130 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-11-19T02:44:20.131 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-11-19T02:44:20.131 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2020-11-19T02:44:20.131 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2020-11-19T02:44:20.132 INFO:teuthology.orchestra.run.smithi086:> sudo apt-get clean 2020-11-19T02:44:20.220 INFO:teuthology.orchestra.run.smithi086:> sudo apt-get update 2020-11-19T02:44:20.344 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-19T02:44:20.455 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T02:44:20.533 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-19T02:44:20.611 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-19T02:44:22.223 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2020-11-19T02:44:23.572 INFO:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-19T02:44:23.655 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2020-11-19T02:44:23.889 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2020-11-19T02:44:23.890 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2020-11-19T02:44:23.969 INFO:teuthology.orchestra.run.smithi086.stderr:E: Unable to locate package linux-image-current-generic 2020-11-19T02:44:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-19T02:44:23.971 INFO:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-19T02:44:24.049 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2020-11-19T02:44:24.275 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2020-11-19T02:44:24.276 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2020-11-19T02:44:24.469 INFO:teuthology.orchestra.run.smithi086.stdout:linux-image-generic is already the newest version (4.15.0.124.111). 2020-11-19T02:44:24.470 INFO:teuthology.orchestra.run.smithi086.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T02:44:24.471 INFO:teuthology.orchestra.run.smithi086:> dpkg -s linux-image-generic 2020-11-19T02:44:24.496 INFO:teuthology.orchestra.run.smithi086.stdout:Package: linux-image-generic 2020-11-19T02:44:24.496 INFO:teuthology.orchestra.run.smithi086.stdout:Status: install ok installed 2020-11-19T02:44:24.496 INFO:teuthology.orchestra.run.smithi086.stdout:Priority: optional 2020-11-19T02:44:24.496 INFO:teuthology.orchestra.run.smithi086.stdout:Section: kernel 2020-11-19T02:44:24.497 INFO:teuthology.orchestra.run.smithi086.stdout:Installed-Size: 16 2020-11-19T02:44:24.497 INFO:teuthology.orchestra.run.smithi086.stdout:Maintainer: Ubuntu Kernel Team 2020-11-19T02:44:24.497 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: amd64 2020-11-19T02:44:24.497 INFO:teuthology.orchestra.run.smithi086.stdout:Source: linux-meta 2020-11-19T02:44:24.498 INFO:teuthology.orchestra.run.smithi086.stdout:Version: 4.15.0.124.111 2020-11-19T02:44:24.498 INFO:teuthology.orchestra.run.smithi086.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-19T02:44:24.498 INFO:teuthology.orchestra.run.smithi086.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-19T02:44:24.498 INFO:teuthology.orchestra.run.smithi086.stdout:Recommends: thermald 2020-11-19T02:44:24.498 INFO:teuthology.orchestra.run.smithi086.stdout:Description: Generic Linux kernel image 2020-11-19T02:44:24.499 INFO:teuthology.orchestra.run.smithi086.stdout: This package will always depend on the latest generic kernel image 2020-11-19T02:44:24.499 INFO:teuthology.orchestra.run.smithi086.stdout: available. 2020-11-19T02:44:24.499 INFO:teuthology.orchestra.run.smithi086:> mktemp 2020-11-19T02:44:24.544 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.nQcsoHjjhc 2020-11-19T02:44:24.545 INFO:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub/grub.cfg /tmp/tmp.nQcsoHjjhc 2020-11-19T02:44:24.611 INFO:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.nQcsoHjjhc 2020-11-19T02:44:24.731 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.nQcsoHjjhc is 8KB 2020-11-19T02:44:24.781 INFO:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.nQcsoHjjhc 2020-11-19T02:44:24.788 INFO:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-11-19T02:44:24.846 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:44:24.847 INFO:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-19T02:44:24.847 INFO:teuthology.orchestra.run.smithi086:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-19T02:44:24.891 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-19T02:44:24.909 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:44:24.910 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:44:24.910 INFO:teuthology.orchestra.run.smithi086.stderr:93 bytes copied, 3.1388e-05 s, 3.0 MB/s 2020-11-19T02:44:24.910 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-19T02:44:24.919 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-124-generic 2020-11-19T02:44:24.919 INFO:teuthology.orchestra.run.smithi086:> sudo update-grub 2020-11-19T02:44:26.286 INFO:teuthology.orchestra.run.smithi086.stderr:Sourcing file `/etc/default/grub' 2020-11-19T02:44:26.308 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2020-11-19T02:44:26.549 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-19T02:44:26.560 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-19T02:44:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-19T02:44:26.868 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-19T02:44:27.450 INFO:teuthology.orchestra.run.smithi086.stderr:done 2020-11-19T02:44:27.452 INFO:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2020-11-19T02:44:27.465 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2020-11-19T02:44:27.466 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2020-11-19T02:44:27.467 INFO:teuthology.orchestra.run.smithi160:> sudo apt-get clean 2020-11-19T02:44:27.543 INFO:teuthology.orchestra.run.smithi160:> sudo apt-get update 2020-11-19T02:44:27.755 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-19T02:44:27.866 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T02:44:27.945 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-19T02:44:28.028 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-19T02:44:29.617 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2020-11-19T02:44:29.665 INFO:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-19T02:44:29.746 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2020-11-19T02:44:29.979 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2020-11-19T02:44:29.980 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2020-11-19T02:44:30.061 INFO:teuthology.orchestra.run.smithi160.stderr:E: Unable to locate package linux-image-current-generic 2020-11-19T02:44:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-19T02:44:30.063 INFO:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-19T02:44:30.142 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2020-11-19T02:44:30.371 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2020-11-19T02:44:30.372 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2020-11-19T02:44:30.567 INFO:teuthology.orchestra.run.smithi160.stdout:linux-image-generic is already the newest version (4.15.0.124.111). 2020-11-19T02:44:30.567 INFO:teuthology.orchestra.run.smithi160.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T02:44:30.569 INFO:teuthology.orchestra.run.smithi160:> dpkg -s linux-image-generic 2020-11-19T02:44:30.589 INFO:teuthology.orchestra.run.smithi160.stdout:Package: linux-image-generic 2020-11-19T02:44:30.590 INFO:teuthology.orchestra.run.smithi160.stdout:Status: install ok installed 2020-11-19T02:44:30.590 INFO:teuthology.orchestra.run.smithi160.stdout:Priority: optional 2020-11-19T02:44:30.590 INFO:teuthology.orchestra.run.smithi160.stdout:Section: kernel 2020-11-19T02:44:30.590 INFO:teuthology.orchestra.run.smithi160.stdout:Installed-Size: 16 2020-11-19T02:44:30.591 INFO:teuthology.orchestra.run.smithi160.stdout:Maintainer: Ubuntu Kernel Team 2020-11-19T02:44:30.591 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: amd64 2020-11-19T02:44:30.591 INFO:teuthology.orchestra.run.smithi160.stdout:Source: linux-meta 2020-11-19T02:44:30.591 INFO:teuthology.orchestra.run.smithi160.stdout:Version: 4.15.0.124.111 2020-11-19T02:44:30.592 INFO:teuthology.orchestra.run.smithi160.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-19T02:44:30.592 INFO:teuthology.orchestra.run.smithi160.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-19T02:44:30.592 INFO:teuthology.orchestra.run.smithi160.stdout:Recommends: thermald 2020-11-19T02:44:30.592 INFO:teuthology.orchestra.run.smithi160.stdout:Description: Generic Linux kernel image 2020-11-19T02:44:30.592 INFO:teuthology.orchestra.run.smithi160.stdout: This package will always depend on the latest generic kernel image 2020-11-19T02:44:30.593 INFO:teuthology.orchestra.run.smithi160.stdout: available. 2020-11-19T02:44:30.593 INFO:teuthology.orchestra.run.smithi160:> mktemp 2020-11-19T02:44:30.636 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.4WkMpUavXd 2020-11-19T02:44:30.636 INFO:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub/grub.cfg /tmp/tmp.4WkMpUavXd 2020-11-19T02:44:30.694 INFO:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.4WkMpUavXd 2020-11-19T02:44:30.810 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.4WkMpUavXd is 8KB 2020-11-19T02:44:30.821 INFO:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.4WkMpUavXd 2020-11-19T02:44:30.828 INFO:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-11-19T02:44:30.885 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:44:30.885 INFO:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-19T02:44:30.886 INFO:teuthology.orchestra.run.smithi160:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-19T02:44:30.931 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-19T02:44:30.945 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:44:30.945 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:44:30.946 INFO:teuthology.orchestra.run.smithi160.stderr:93 bytes copied, 3.132e-05 s, 3.0 MB/s 2020-11-19T02:44:30.946 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-19T02:44:30.955 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-124-generic 2020-11-19T02:44:30.956 INFO:teuthology.orchestra.run.smithi160:> sudo update-grub 2020-11-19T02:44:32.192 INFO:teuthology.orchestra.run.smithi160.stderr:Sourcing file `/etc/default/grub' 2020-11-19T02:44:32.210 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2020-11-19T02:44:32.474 INFO:teuthology.orchestra.run.smithi160.stderr:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-19T02:44:32.486 INFO:teuthology.orchestra.run.smithi160.stderr:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-19T02:44:32.798 INFO:teuthology.orchestra.run.smithi160.stderr:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-19T02:44:32.811 INFO:teuthology.orchestra.run.smithi160.stderr:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-19T02:44:33.790 INFO:teuthology.orchestra.run.smithi160.stderr:done 2020-11-19T02:44:33.792 INFO:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2020-11-19T02:45:03.827 INFO:teuthology.misc:Re-opening connections... 2020-11-19T02:45:03.827 INFO:teuthology.misc:trying to connect to ubuntu@smithi086.front.sepia.ceph.com 2020-11-19T02:45:03.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T02:45:03.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:46:03.830 DEBUG:teuthology.orchestra.remote:timed out 2020-11-19T02:46:03.911 INFO:teuthology.misc:trying to connect to ubuntu@smithi160.front.sepia.ceph.com 2020-11-19T02:46:03.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T02:46:03.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:46:22.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2020-11-19T02:46:22.231 DEBUG:teuthology.misc:waited 78.40340876579285 2020-11-19T02:46:23.231 INFO:teuthology.misc:trying to connect to ubuntu@smithi086.front.sepia.ceph.com 2020-11-19T02:46:23.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T02:46:23.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:46:23.375 INFO:teuthology.orchestra.run.smithi086:> true 2020-11-19T02:46:23.700 DEBUG:teuthology.misc:waited 79.87315583229065 2020-11-19T02:46:24.702 INFO:teuthology.misc:trying to connect to ubuntu@smithi160.front.sepia.ceph.com 2020-11-19T02:46:24.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T02:46:24.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:46:24.831 INFO:teuthology.orchestra.run.smithi160:> true 2020-11-19T02:46:25.454 DEBUG:teuthology.misc:waited 81.6271960735321 2020-11-19T02:46:26.456 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2020-11-19T02:46:26.457 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-124-generic"... 2020-11-19T02:46:26.457 INFO:teuthology.orchestra.run.smithi086:> uname -r 2020-11-19T02:46:26.471 INFO:teuthology.orchestra.run.smithi086.stdout:4.15.0-124-generic 2020-11-19T02:46:26.472 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-124-generic vs 4.15.0-124-generic 2020-11-19T02:46:26.473 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-11-19T02:46:26.474 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-11-19T02:46:26.476 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2020-11-19T02:46:26.478 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-124-generic"... 2020-11-19T02:46:26.479 INFO:teuthology.orchestra.run.smithi160:> uname -r 2020-11-19T02:46:26.489 INFO:teuthology.orchestra.run.smithi160.stdout:4.15.0-124-generic 2020-11-19T02:46:26.490 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-124-generic vs 4.15.0-124-generic 2020-11-19T02:46:26.490 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-11-19T02:46:26.491 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-11-19T02:46:27.492 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-11-19T02:46:27.492 INFO:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-19T02:46:27.547 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2020-11-19T02:46:27.573 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-11-19T02:46:27.574 INFO:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-19T02:46:27.914 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2020-11-19T02:46:27.938 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-19T02:46:27.945 INFO:teuthology.task.internal:Creating test directory... 2020-11-19T02:46:27.945 INFO:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-19T02:46:27.948 INFO:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-19T02:46:27.954 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-19T02:46:27.970 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-19T02:46:28.075 INFO:teuthology.task.internal:Creating archive directory... 2020-11-19T02:46:28.075 INFO:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-19T02:46:28.084 INFO:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-19T02:46:28.112 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-19T02:46:28.130 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-19T02:46:28.131 INFO:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-19T02:46:28.159 INFO:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-19T02:46:28.199 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-19T02:46:28.210 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-19T02:46:28.214 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-19T02:46:28.224 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-19T02:46:28.225 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-19T02:46:28.254 INFO:teuthology.task.internal:Configuring sudo... 2020-11-19T02:46:28.255 INFO:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-19T02:46:28.284 INFO:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-19T02:46:28.313 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-19T02:46:28.384 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-19T02:46:28.385 INFO:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-19T02:46:28.392 INFO:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-19T02:46:28.403 INFO:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-19T02:46:28.453 INFO:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-19T02:46:28.507 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:46:28.508 INFO:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-19T02:46:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-19T02:46:28.611 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:46:28.611 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:46:28.612 INFO:teuthology.orchestra.run.smithi086.stderr:152 bytes copied, 0.029829 s, 5.1 kB/s 2020-11-19T02:46:28.613 INFO:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-19T02:46:28.628 INFO:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-19T02:46:28.678 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:46:28.679 INFO:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-19T02:46:28.732 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-19T02:46:28.781 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:46:28.782 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:46:28.783 INFO:teuthology.orchestra.run.smithi160.stderr:152 bytes copied, 0.0295523 s, 5.1 kB/s 2020-11-19T02:46:28.786 INFO:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2020-11-19T02:46:28.795 INFO:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2020-11-19T02:46:28.879 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-19T02:46:28.896 INFO:teuthology.task.internal:Starting timer... 2020-11-19T02:46:28.897 INFO:teuthology.run_tasks:Running task pcp... 2020-11-19T02:46:29.023 INFO:teuthology.run_tasks:Running task selinux... 2020-11-19T02:46:29.082 DEBUG:teuthology.task.selinux:Excluding smithi086: OS 'ubuntu' does not support SELinux 2020-11-19T02:46:29.083 DEBUG:teuthology.task.selinux:Excluding smithi160: OS 'ubuntu' does not support SELinux 2020-11-19T02:46:29.083 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-19T02:46:29.083 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-11-19T02:46:29.084 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-19T02:46:29.084 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-19T02:46:29.111 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-11-19T02:46:29.112 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-19T02:46:29.134 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-19T02:46:29.135 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-19T02:46:29.634 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-19T02:46:31.324 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-19T02:46:34.079 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-19T02:46:34.106 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:34 +0000 (0:00:00.146) 0:00:00.146 ***** 2020-11-19T02:46:34.120 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:34 +0000 (0:00:00.014) 0:00:00.160 ***** 2020-11-19T02:46:35.819 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi086.front.sepia.ceph.com] 2020-11-19T02:46:35.925 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:35 +0000 (0:00:01.802) 0:00:01.963 ***** 2020-11-19T02:46:35.937 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] 2020-11-19T02:46:36.048 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:36 +0000 (0:00:00.124) 0:00:02.087 ***** 2020-11-19T02:46:36.193 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-11-19T02:46:36.195 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-19T02:46:36.229 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:36 +0000 (0:00:00.181) 0:00:02.269 ***** 2020-11-19T02:46:36.268 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:36 +0000 (0:00:00.039) 0:00:02.309 ***** 2020-11-19T02:46:36.646 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-19T02:46:36.648 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-19T02:46:36.681 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:36 +0000 (0:00:00.413) 0:00:02.722 ***** 2020-11-19T02:46:36.700 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-19T02:46:36.738 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:36 +0000 (0:00:00.054) 0:00:02.776 ***** 2020-11-19T02:46:37.013 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.093393", "end": "2020-11-19 02:46:37.141876", "rc": 0, "start": "2020-11-19 02:46:37.048483", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-19T02:46:37.056 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:37 +0000 (0:00:00.320) 0:00:03.097 ***** 2020-11-19T02:46:37.065 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.094374", "end": "2020-11-19 02:46:36.741557", "rc": 0, "start": "2020-11-19 02:46:36.647183", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-19T02:46:37.114 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:37 +0000 (0:00:00.054) 0:00:03.151 ***** 2020-11-19T02:46:37.301 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi160.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T02:46:37.336 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:37 +0000 (0:00:00.225) 0:00:03.377 ***** 2020-11-19T02:46:37.342 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T02:46:37.383 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:37 +0000 (0:00:00.046) 0:00:03.423 ***** 2020-11-19T02:46:38.177 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-19T02:46:38.179 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605753997.397261-274334131410246/source", "state": "file", "uid": 0} changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605753997.3487449-277240865270539/source", "state": "file", "uid": 0} 2020-11-19T02:46:38.212 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:38 +0000 (0:00:00.829) 0:00:04.253 ***** 2020-11-19T02:46:38.245 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:38 +0000 (0:00:00.032) 0:00:04.285 ***** 2020-11-19T02:46:38.572 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-19T02:46:38.574 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-19T02:46:38.576 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-19T02:46:38.645 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-19T02:46:38.647 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-19T02:46:38.741 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-19T02:46:38.745 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-19T02:46:38.900 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:38 +0000 (0:00:00.654) 0:00:04.940 ***** 2020-11-19T02:46:38.934 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:38 +0000 (0:00:00.034) 0:00:04.974 ***** 2020-11-19T02:46:39.519 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-11-19T02:46:39.954 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Thursday 19 November 2020 02:46:39 +0000 (0:00:00.692) 0:00:05.667 ***** ok: [smithi160.front.sepia.ceph.com] Thursday 19 November 2020 02:46:39 +0000 (0:00:00.127) 0:00:05.795 ***** 2020-11-19T02:46:40.057 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-19T02:46:40.059 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.213944", "end": "2020-11-19 02:46:40.025418", "rc": 0, "start": "2020-11-19 02:46:39.811474", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-19T02:46:40.094 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.339) 0:00:06.134 ***** 2020-11-19T02:46:40.123 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.169676", "end": "2020-11-19 02:46:40.096134", "rc": 0, "start": "2020-11-19 02:46:39.926458", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-19T02:46:40.172 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.077) 0:00:06.212 ***** 2020-11-19T02:46:40.241 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-19T02:46:40.243 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008205", "end": "2020-11-19 02:46:40.366749", "failed_when_result": false, "rc": 0, "start": "2020-11-19 02:46:40.358544", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-19T02:46:40.282 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.110) 0:00:06.323 ***** 2020-11-19T02:46:40.320 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-19T02:46:40.321 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.020545", "end": "2020-11-19 02:46:40.040146", "failed_when_result": false, "rc": 0, "start": "2020-11-19 02:46:40.019601", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-19T02:46:40.367 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.085) 0:00:06.408 ***** 2020-11-19T02:46:40.410 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.042) 0:00:06.451 ***** 2020-11-19T02:46:40.416 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-19T02:46:40.418 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.462 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.051) 0:00:06.503 ***** 2020-11-19T02:46:40.472 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-19T02:46:40.474 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-19T02:46:40.518 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.055) 0:00:06.558 ***** 2020-11-19T02:46:40.524 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-11-19T02:46:40.525 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.560 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.042) 0:00:06.601 ***** 2020-11-19T02:46:40.567 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-19T02:46:40.569 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.603 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.042) 0:00:06.644 ***** 2020-11-19T02:46:40.609 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-19T02:46:40.611 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.646 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.042) 0:00:06.686 ***** 2020-11-19T02:46:40.652 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-11-19T02:46:40.654 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.688 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.042) 0:00:06.728 ***** 2020-11-19T02:46:40.694 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-19T02:46:40.696 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.732 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.043) 0:00:06.772 ***** 2020-11-19T02:46:40.742 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-19T02:46:40.744 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.795 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.063) 0:00:06.836 ***** 2020-11-19T02:46:40.804 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-19T02:46:40.806 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.841 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.046) 0:00:06.882 ***** 2020-11-19T02:46:40.848 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-19T02:46:40.850 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.885 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.043) 0:00:06.926 ***** 2020-11-19T02:46:40.902 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-19T02:46:40.903 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.914 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-19T02:46:40.915 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.952 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:40 +0000 (0:00:00.067) 0:00:06.993 ***** 2020-11-19T02:46:40.961 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.962 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.964 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.964 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.965 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.966 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.967 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.968 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B93 2020-11-19T02:46:40.969 INFO:teuthology.task.ansible.out:5dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7Y 2020-11-19T02:46:40.971 INFO:teuthology.task.ansible.out:yYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.980 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.982 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.991 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:40.993 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.003 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.005 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.014 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.016 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.026 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.028 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.038 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.040 INFO:teuthology.task.ansible.out: skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.041 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd 2020-11-19T02:46:41.043 INFO:teuthology.task.ansible.out:0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.052 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.054 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.065 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.067 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.078 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.080 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.091 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.103 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.115 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naeme 2020-11-19T02:46:41.117 INFO:teuthology.task.ansible.out:rson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a0048 2020-11-19T02:46:41.121 INFO:teuthology.task.ansible.out:1a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.130 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.132 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79 2020-11-19T02:46:41.134 INFO:teuthology.task.ansible.out:c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.150 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.161 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.172 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.174 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": 2020-11-19T02:46:41.176 INFO:teuthology.task.ansible.out: "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.186 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.189 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.191 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.192 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n 2020-11-19T02:46:41.193 INFO:teuthology.task.ansible.out:'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.203 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.205 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] 2020-11-19T02:46:41.206 INFO:teuthology.task.ansible.out:=> (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.217 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.218 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.228 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.230 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.241 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.243 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.254 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.256 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2 2020-11-19T02:46:41.257 INFO:teuthology.task.ansible.out:d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.267 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.268 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.281 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.283 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.285 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.287 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.299 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.301 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.314 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.316 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.316 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.317 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMr 2020-11-19T02:46:41.318 INFO:teuthology.task.ansible.out:qJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.328 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.330 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovp 2020-11-19T02:46:41.332 INFO:teuthology.task.ansible.out:n": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.343 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.355 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.357 INFO:teuthology.task.ansible.out: skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.357 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.370 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional resu 2020-11-19T02:46:41.372 INFO:teuthology.task.ansible.out:lt was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.383 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.385 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.395 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.397 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.409 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.411 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.412 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.414 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.429 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.430 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.432 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.434 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.435 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.437 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.438 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.440 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.441 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.452 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.454 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.464 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.467 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.478 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.480 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.481 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.481 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.495 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.497 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.499 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.511 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.512 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.525 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.527 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.536 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.538 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.549 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.550 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.551 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.553 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.565 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.566 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.567 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.567 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.578 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.580 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.590 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.592 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.603 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.606 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.616 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.618 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.629 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.630 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.641 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.643 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.654 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.657 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.666 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.668 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.679 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.681 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.697 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.699 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.734 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:41 +0000 (0:00:00.782) 0:00:07.775 ***** 2020-11-19T02:46:41.742 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.744 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.745 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] = 2020-11-19T02:46:41.746 INFO:teuthology.task.ansible.out:> (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.756 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.758 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} TASK [users : Create all users without sudo access.] *************************** TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.799 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:41 +0000 (0:00:00.063) 0:00:07.839 ***** 2020-11-19T02:46:41.809 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.810 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.812 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.812 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.812 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.814 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.816 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.816 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.827 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.831 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.871 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:41 +0000 (0:00:00.073) 0:00:07.912 ***** 2020-11-19T02:46:41.879 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.881 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.882 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.882 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "o 2020-11-19T02:46:41.884 INFO:teuthology.task.ansible.out:vpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.894 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.896 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.907 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-19T02:46:41.908 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.944 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:41 +0000 (0:00:00.072) 0:00:07.984 ***** 2020-11-19T02:46:41.952 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.954 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.954 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.955 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansib 2020-11-19T02:46:41.956 INFO:teuthology.task.ansible.out:le_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.966 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.968 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:41.983 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.018 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.073) 0:00:08.058 ***** 2020-11-19T02:46:42.051 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.033) 0:00:08.092 ***** 2020-11-19T02:46:42.058 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-19T02:46:42.061 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.095 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.043) 0:00:08.136 ***** 2020-11-19T02:46:42.102 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-19T02:46:42.138 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.042) 0:00:08.178 ***** 2020-11-19T02:46:42.145 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.181 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.043) 0:00:08.222 ***** 2020-11-19T02:46:42.187 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-19T02:46:42.189 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.234 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.052) 0:00:08.274 ***** 2020-11-19T02:46:42.240 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-19T02:46:42.243 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.244 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.256 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.258 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi086.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.295 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.061) 0:00:08.336 ***** 2020-11-19T02:46:42.302 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.304 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.304 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.305 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.316 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.318 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.328 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-19T02:46:42.330 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi086.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.370 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.073) 0:00:08.410 ***** 2020-11-19T02:46:42.376 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.378 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.378 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.379 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.380 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.381 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.381 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.393 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.395 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.431 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.061) 0:00:08.471 ***** 2020-11-19T02:46:42.437 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-19T02:46:42.439 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.439 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.440 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.441 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.442 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.453 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.455 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi086.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.495 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.064) 0:00:08.536 ***** 2020-11-19T02:46:42.531 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.035) 0:00:08.571 ***** 2020-11-19T02:46:42.575 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.577 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.588 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.589 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.606 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.617 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.619 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.631 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.642 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.655 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.666 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.668 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.680 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.689 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.691 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.701 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.703 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.712 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.714 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.724 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.725 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.736 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.738 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.748 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.748 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.759 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.761 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.771 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.772 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.782 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.784 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.795 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.796 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.798 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.808 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.820 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.822 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:42.872 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:42 +0000 (0:00:00.341) 0:00:08.913 ***** 2020-11-19T02:46:43.109 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-19T02:46:43.111 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-19T02:46:43.147 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.274) 0:00:09.187 ***** 2020-11-19T02:46:43.174 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-19T02:46:43.176 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => { "msg": "Host smithi086.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-19T02:46:43.211 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.063) 0:00:09.251 ***** 2020-11-19T02:46:43.219 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-19T02:46:43.261 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.047) 0:00:09.299 ***** 2020-11-19T02:46:43.265 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-19T02:46:43.267 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-19T02:46:43.305 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.045) 0:00:09.344 ***** 2020-11-19T02:46:43.310 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-19T02:46:43.316 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => { "msg": "Host smithi160.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-19T02:46:43.350 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.045) 0:00:09.390 ***** 2020-11-19T02:46:43.358 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.394 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.044) 0:00:09.435 ***** 2020-11-19T02:46:43.403 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-19T02:46:43.405 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-19T02:46:43.448 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.053) 0:00:09.488 ***** 2020-11-19T02:46:43.454 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-19T02:46:43.456 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.491 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.043) 0:00:09.531 ***** 2020-11-19T02:46:43.497 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-19T02:46:43.499 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.534 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.043) 0:00:09.575 ***** 2020-11-19T02:46:43.540 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-19T02:46:43.541 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.578 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.043) 0:00:09.618 ***** 2020-11-19T02:46:43.584 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-19T02:46:43.586 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.621 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.043) 0:00:09.662 ***** 2020-11-19T02:46:43.627 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-19T02:46:43.630 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.664 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.042) 0:00:09.704 ***** 2020-11-19T02:46:43.670 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-19T02:46:43.672 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.706 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.041) 0:00:09.746 ***** 2020-11-19T02:46:43.713 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-19T02:46:43.715 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.749 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.043) 0:00:09.789 ***** 2020-11-19T02:46:43.755 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-19T02:46:43.757 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.793 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.044) 0:00:09.834 ***** 2020-11-19T02:46:43.800 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-19T02:46:43.802 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.837 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.043) 0:00:09.877 ***** 2020-11-19T02:46:43.843 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-19T02:46:43.846 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.887 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.050) 0:00:09.928 ***** 2020-11-19T02:46:43.897 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-19T02:46:43.899 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.944 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.057) 0:00:09.985 ***** 2020-11-19T02:46:43.955 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-19T02:46:43.957 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:43.992 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:43 +0000 (0:00:00.048) 0:00:10.033 ***** 2020-11-19T02:46:44.011 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-19T02:46:44.015 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.049 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.056) 0:00:10.089 ***** 2020-11-19T02:46:44.056 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-19T02:46:44.059 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.103 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.054) 0:00:10.144 ***** 2020-11-19T02:46:44.112 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-19T02:46:44.114 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:44.157 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.053) 0:00:10.197 ***** 2020-11-19T02:46:44.481 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 02:46:44 +0000 (0:00:00.055) 0:00:10.253 ***** TASK [common : Set have_entitlements] ****************************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 02:46:44 +0000 (0:00:00.046) 0:00:10.299 ***** TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi160.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi160.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Thursday 19 November 2020 02:46:44 +0000 (0:00:00.043) 0:00:10.343 ***** TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 02:46:44 +0000 (0:00:00.042) 0:00:10.385 ***** TASK [common : Set have_entitlements] ****************************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 02:46:44 +0000 (0:00:00.067) 0:00:10.453 ***** TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Thursday 19 November 2020 02:46:44 +0000 (0:00:00.047) 0:00:10.500 ***** TASK [common 2020-11-19T02:46:44.483 INFO:teuthology.task.ansible.out: : Install CA Cert from Satellite Server] ************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.502 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.042) 0:00:10.543 ***** 2020-11-19T02:46:44.508 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-19T02:46:44.510 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.547 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.044) 0:00:10.587 ***** 2020-11-19T02:46:44.554 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-19T02:46:44.556 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:44.593 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.045) 0:00:10.632 ***** 2020-11-19T02:46:44.598 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-19T02:46:44.600 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:44.635 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.043) 0:00:10.676 ***** 2020-11-19T02:46:44.643 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-19T02:46:44.645 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.679 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.044) 0:00:10.720 ***** 2020-11-19T02:46:44.685 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-19T02:46:44.687 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.727 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.047) 0:00:10.768 ***** 2020-11-19T02:46:44.734 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi160.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:44.772 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.044) 0:00:10.812 ***** 2020-11-19T02:46:44.779 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-19T02:46:44.781 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.834 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.062) 0:00:10.875 ***** 2020-11-19T02:46:44.843 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-19T02:46:44.844 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.880 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.045) 0:00:10.921 ***** 2020-11-19T02:46:44.887 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.923 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.042) 0:00:10.963 ***** 2020-11-19T02:46:44.930 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-19T02:46:44.932 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:44.966 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:44 +0000 (0:00:00.043) 0:00:11.007 ***** 2020-11-19T02:46:44.972 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-19T02:46:44.973 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.011 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.044) 0:00:11.051 ***** 2020-11-19T02:46:45.017 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-19T02:46:45.020 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.053 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.042) 0:00:11.094 ***** 2020-11-19T02:46:45.059 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-19T02:46:45.066 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:45.097 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.044) 0:00:11.138 ***** 2020-11-19T02:46:45.106 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-19T02:46:45.108 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.156 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.057) 0:00:11.196 ***** 2020-11-19T02:46:45.164 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.207 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.050) 0:00:11.247 ***** 2020-11-19T02:46:45.213 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-19T02:46:45.215 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:45.250 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.043) 0:00:11.290 ***** 2020-11-19T02:46:45.256 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-19T02:46:45.258 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.294 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.043) 0:00:11.334 ***** 2020-11-19T02:46:45.301 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-19T02:46:45.302 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.338 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.043) 0:00:11.378 ***** 2020-11-19T02:46:45.344 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-19T02:46:45.346 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.381 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.043) 0:00:11.422 ***** 2020-11-19T02:46:45.394 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-19T02:46:45.396 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.429 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.048) 0:00:11.470 ***** 2020-11-19T02:46:45.435 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-19T02:46:45.437 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.472 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.042) 0:00:11.513 ***** 2020-11-19T02:46:45.482 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.515 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.042) 0:00:11.556 ***** 2020-11-19T02:46:45.521 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-19T02:46:45.522 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.557 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.042) 0:00:11.598 ***** 2020-11-19T02:46:45.563 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-19T02:46:45.565 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.600 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.042) 0:00:11.641 ***** 2020-11-19T02:46:45.606 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-19T02:46:45.608 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.642 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.042) 0:00:11.683 ***** 2020-11-19T02:46:45.650 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.687 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.044) 0:00:11.727 ***** 2020-11-19T02:46:45.693 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-19T02:46:45.694 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:45.730 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.042) 0:00:11.770 ***** 2020-11-19T02:46:45.736 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-19T02:46:45.738 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.776 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.045) 0:00:11.816 ***** 2020-11-19T02:46:45.782 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-19T02:46:45.784 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.823 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.047) 0:00:11.864 ***** 2020-11-19T02:46:45.830 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi160.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:45.867 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.044) 0:00:11.908 ***** 2020-11-19T02:46:45.873 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-19T02:46:45.875 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.910 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.042) 0:00:11.951 ***** 2020-11-19T02:46:45.918 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-19T02:46:45.920 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:45.959 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.048) 0:00:11.999 ***** 2020-11-19T02:46:45.965 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-19T02:46:45.969 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.002 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:45 +0000 (0:00:00.043) 0:00:12.043 ***** 2020-11-19T02:46:46.009 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-19T02:46:46.011 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.050 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.047) 0:00:12.090 ***** 2020-11-19T02:46:46.056 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-19T02:46:46.058 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.094 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.044) 0:00:12.135 ***** 2020-11-19T02:46:46.103 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-19T02:46:46.104 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.142 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.048) 0:00:12.183 ***** 2020-11-19T02:46:46.149 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-19T02:46:46.151 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.201 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.058) 0:00:12.242 ***** 2020-11-19T02:46:46.209 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-19T02:46:46.212 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.252 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.051) 0:00:12.293 ***** 2020-11-19T02:46:46.259 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-19T02:46:46.261 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.303 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.050) 0:00:12.344 ***** 2020-11-19T02:46:46.310 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-19T02:46:46.312 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.354 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.050) 0:00:12.394 ***** 2020-11-19T02:46:46.380 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-19T02:46:46.382 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.416 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.062) 0:00:12.456 ***** 2020-11-19T02:46:46.455 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-19T02:46:46.457 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.018202", "end": "2020-11-19 02:46:46.574228", "rc": 0, "start": "2020-11-19 02:46:46.556026", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-19T02:46:46.491 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.075) 0:00:12.532 ***** 2020-11-19T02:46:46.519 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-19T02:46:46.521 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.554 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.063) 0:00:12.595 ***** 2020-11-19T02:46:46.581 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-19T02:46:46.583 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.617 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.062) 0:00:12.658 ***** 2020-11-19T02:46:46.644 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.680 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.063) 0:00:12.721 ***** 2020-11-19T02:46:46.698 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-19T02:46:46.700 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.153483", "end": "2020-11-19 02:46:46.417505", "rc": 0, "start": "2020-11-19 02:46:46.264022", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-19T02:46:46.734 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.053) 0:00:12.774 ***** 2020-11-19T02:46:46.740 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-19T02:46:46.742 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.781 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.047) 0:00:12.822 ***** 2020-11-19T02:46:46.788 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-19T02:46:46.790 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.825 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.043) 0:00:12.866 ***** 2020-11-19T02:46:46.832 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-19T02:46:46.834 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.873 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.048) 0:00:12.914 ***** 2020-11-19T02:46:46.880 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-19T02:46:46.882 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.917 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.043) 0:00:12.958 ***** 2020-11-19T02:46:46.924 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-19T02:46:46.925 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:46.961 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.043) 0:00:13.002 ***** 2020-11-19T02:46:46.968 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-19T02:46:46.969 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:47.005 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:46 +0000 (0:00:00.044) 0:00:13.046 ***** 2020-11-19T02:46:47.011 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-19T02:46:47.013 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:47.049 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.043) 0:00:13.090 ***** 2020-11-19T02:46:47.055 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-19T02:46:47.058 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:47.096 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.047) 0:00:13.137 ***** 2020-11-19T02:46:47.124 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-19T02:46:47.126 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:47.159 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.062) 0:00:13.199 ***** 2020-11-19T02:46:47.186 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-19T02:46:47.188 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:47.223 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.063) 0:00:13.263 ***** 2020-11-19T02:46:47.250 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-19T02:46:47.252 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:47.285 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.061) 0:00:13.325 ***** 2020-11-19T02:46:47.376 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-19T02:46:47.378 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.205089", "end": "2020-11-19 02:46:47.500318", "rc": 0, "start": "2020-11-19 02:46:47.295229", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:46:47.417 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.132) 0:00:13.458 ***** 2020-11-19T02:46:47.445 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-19T02:46:47.446 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:47.480 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.062) 0:00:13.521 ***** 2020-11-19T02:46:47.675 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-19T02:46:47.676 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.033159", "end": "2020-11-19 02:46:47.789729", "rc": 0, "start": "2020-11-19 02:46:47.756570", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:46:47.716 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.236) 0:00:13.757 ***** 2020-11-19T02:46:47.775 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-19T02:46:47.777 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.375869", "end": "2020-11-19 02:46:47.500310", "rc": 0, "start": "2020-11-19 02:46:47.124441", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:46:47.815 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.098) 0:00:13.856 ***** 2020-11-19T02:46:47.843 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-19T02:46:47.844 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:47.882 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:47 +0000 (0:00:00.066) 0:00:13.922 ***** 2020-11-19T02:46:48.042 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-19T02:46:48.043 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.030874", "end": "2020-11-19 02:46:47.753147", "rc": 0, "start": "2020-11-19 02:46:47.722273", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:46:48.080 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:48 +0000 (0:00:00.198) 0:00:14.121 ***** 2020-11-19T02:46:51.439 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-19T02:46:51.441 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:46:51.475 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:51 +0000 (0:00:03.395) 0:00:17.516 ***** 2020-11-19T02:46:51.482 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:46:51.519 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:51 +0000 (0:00:00.043) 0:00:17.560 ***** 2020-11-19T02:46:52.375 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-19T02:46:52.377 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:46:52.413 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:52 +0000 (0:00:00.893) 0:00:18.453 ***** 2020-11-19T02:46:52.419 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:46:52.458 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:52 +0000 (0:00:00.045) 0:00:18.498 ***** 2020-11-19T02:46:52.464 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-19T02:46:52.466 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:52.503 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:52 +0000 (0:00:00.045) 0:00:18.544 ***** 2020-11-19T02:46:52.510 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:52.548 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:52 +0000 (0:00:00.045) 0:00:18.589 ***** 2020-11-19T02:46:52.849 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-19T02:46:52.887 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:52 +0000 (0:00:00.338) 0:00:18.927 ***** 2020-11-19T02:46:52.894 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-19T02:46:52.934 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:52 +0000 (0:00:00.047) 0:00:18.974 ***** 2020-11-19T02:46:53.704 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:46:53.739 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:53 +0000 (0:00:00.804) 0:00:19.779 ***** 2020-11-19T02:46:53.782 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:46:53.824 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:53 +0000 (0:00:00.080) 0:00:19.860 ***** 2020-11-19T02:46:54.107 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-19T02:46:54.109 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-19T02:46:54.237 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-19T02:46:54.399 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-19T02:46:54.537 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-19T02:46:54.664 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-19T02:46:54.702 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:54 +0000 (0:00:00.882) 0:00:20.742 ***** 2020-11-19T02:46:54.819 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-19T02:46:54.821 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-19T02:46:54.857 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:54 +0000 (0:00:00.154) 0:00:20.897 ***** 2020-11-19T02:46:54.864 INFO:teuthology.task.ansible.out: 2020-11-19T02:46:54.896 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:54 +0000 (0:00:00.038) 0:00:20.936 ***** 2020-11-19T02:46:55.022 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-19T02:46:55.023 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-19T02:46:55.059 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:55 +0000 (0:00:00.162) 0:00:21.099 ***** 2020-11-19T02:46:55.412 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-19T02:46:55.414 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-19T02:46:55.448 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-19T02:46:55.704 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-19T02:46:55.748 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-19T02:46:55.994 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-19T02:46:56.033 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.973) 0:00:22.072 ***** 2020-11-19T02:46:56.041 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-19T02:46:56.081 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.049) 0:00:22.122 ***** 2020-11-19T02:46:56.089 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-19T02:46:56.091 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-19T02:46:56.129 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.047) 0:00:22.169 ***** 2020-11-19T02:46:56.137 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-19T02:46:56.173 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.044) 0:00:22.213 ***** 2020-11-19T02:46:56.179 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-19T02:46:56.181 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:56.216 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.043) 0:00:22.257 ***** 2020-11-19T02:46:56.223 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:56.265 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.048) 0:00:22.306 ***** 2020-11-19T02:46:56.272 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-19T02:46:56.273 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:56.321 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.055) 0:00:22.361 ***** 2020-11-19T02:46:56.329 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:56.379 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.058) 0:00:22.420 ***** 2020-11-19T02:46:56.385 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-19T02:46:56.387 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:56.451 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.072) 0:00:22.492 ***** 2020-11-19T02:46:56.459 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:56.461 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-19T02:46:56.501 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.049) 0:00:22.542 ***** 2020-11-19T02:46:56.508 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-19T02:46:56.510 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:56.558 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.055) 0:00:22.597 ***** 2020-11-19T02:46:56.565 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:56.618 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:56 +0000 (0:00:00.060) 0:00:22.658 ***** 2020-11-19T02:46:57.413 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-19T02:46:57.415 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:46:57.449 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:57 +0000 (0:00:00.832) 0:00:23.490 ***** 2020-11-19T02:46:57.519 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:46:57.556 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:57 +0000 (0:00:00.106) 0:00:23.597 ***** 2020-11-19T02:46:57.787 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-19T02:46:57.790 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-19T02:46:57.833 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:57 +0000 (0:00:00.276) 0:00:23.873 ***** 2020-11-19T02:46:57.883 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-19T02:46:57.920 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:57 +0000 (0:00:00.086) 0:00:23.960 ***** 2020-11-19T02:46:58.184 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-19T02:46:58.186 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-19T02:46:58.232 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.311) 0:00:24.272 ***** 2020-11-19T02:46:58.282 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-19T02:46:58.327 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.094) 0:00:24.367 ***** 2020-11-19T02:46:58.375 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-19T02:46:58.377 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-19T02:46:58.474 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-19T02:46:58.487 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-19T02:46:58.525 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.197) 0:00:24.565 ***** 2020-11-19T02:46:58.555 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-19T02:46:58.556 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:58.596 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.071) 0:00:24.637 ***** 2020-11-19T02:46:58.604 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-19T02:46:58.606 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-19T02:46:58.643 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.046) 0:00:24.683 ***** 2020-11-19T02:46:58.649 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-19T02:46:58.651 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:58.687 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.044) 0:00:24.728 ***** 2020-11-19T02:46:58.694 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:58.731 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.044) 0:00:24.772 ***** 2020-11-19T02:46:58.738 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-19T02:46:58.740 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:58.776 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.044) 0:00:24.816 ***** 2020-11-19T02:46:58.786 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi160.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T02:46:58.820 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.043) 0:00:24.860 ***** 2020-11-19T02:46:58.847 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:58.882 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:58 +0000 (0:00:00.062) 0:00:24.922 ***** 2020-11-19T02:46:59.098 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-19T02:46:59.100 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-19T02:46:59.138 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:59 +0000 (0:00:00.254) 0:00:25.177 ***** 2020-11-19T02:46:59.210 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-19T02:46:59.254 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:59 +0000 (0:00:00.117) 0:00:25.294 ***** 2020-11-19T02:46:59.772 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-19T02:46:59.773 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 02:46:11 UTC", "ActiveEnterTimestampMonotonic": "20397142", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket basic.target systemd-tmpfiles-setup.service sysinit.target nss-lookup.target network.target -.mount remote-fs.target var-run.mount time-sync.target local-fs.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 02:46:11 UTC", "AssertTimestampMonotonic": "20395241", "Before": "xdm.service multi-user.target plymouth-quit.service getty@tty1.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 02:46:11 UTC", "ConditionTimestampMonotonic": "20395241", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-19T02:46:59.774 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1141", "ExecMainStartTimestamp": "Thu 2020-11-19 02:46:11 UTC", "ExecMainStartTimestampMonotonic": "20397075", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-11-19 02:46:11 UTC] ; stop_time=[n/a] ; pid=1141 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 02:46:11 UTC", "InactiveExitTimestampMonotonic": "20397142", "InvocationID": "3755cb351fe543419e4c32442a3c2c17", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-11-19T02:46:59.776 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1141", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-11-19T02:46:59.777 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 02:46:11 UTC", "StateChangeTimestampMonotonic": "20397142", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 02:46:11 UTC", "WatchdogTimestampMonotonic": "20397140", "WatchdogUSec": "0"}} 2020-11-19T02:46:59.809 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:59 +0000 (0:00:00.555) 0:00:25.850 ***** 2020-11-19T02:46:59.818 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 02:46:23 UTC", "ActiveEnterTimestampMonotonic": "20660457", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount remote-fs.target time-sync.target systemd-journald.socket system.slice nss-lookup.target sysinit.target network.target local-fs.target -.mount basic.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 02:46:23 UTC", "AssertTimestampMonotonic": "20658663", "Before": "multi-user.target plymouth-quit.service shutdown.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 02:46:23 UTC", "ConditionTimestampMonotonic": "20658663", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-19T02:46:59.819 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1294", "ExecMainStartTimestamp": "Thu 2020-11-19 02:46:23 UTC", "ExecMainStartTimestampMonotonic": "20660401", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-11-19 02:46:23 UTC] ; stop_time=[n/a] ; pid=1294 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 02:46:23 UTC", "InactiveExitTimestampMonotonic": "20660457", "InvocationID": "c8ab05b59ebc4d3b9521008ca765e42c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-11-19T02:46:59.821 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1294", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-11-19T02:46:59.823 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 02:46:23 UTC", "StateChangeTimestampMonotonic": "20660457", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 02:46:23 UTC", "WatchdogTimestampMonotonic": "20660456", "WatchdogUSec": "0"}} 2020-11-19T02:46:59.854 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:59 +0000 (0:00:00.044) 0:00:25.894 ***** 2020-11-19T02:46:59.861 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-11-19T02:46:59.863 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:59.896 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:59 +0000 (0:00:00.042) 0:00:25.937 ***** 2020-11-19T02:46:59.903 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:59.944 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:59 +0000 (0:00:00.047) 0:00:25.984 ***** 2020-11-19T02:46:59.950 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:46:59.988 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:46:59 +0000 (0:00:00.044) 0:00:26.029 ***** 2020-11-19T02:46:59.997 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.035 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.046) 0:00:26.075 ***** 2020-11-19T02:47:00.042 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-19T02:47:00.043 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.079 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.044) 0:00:26.120 ***** 2020-11-19T02:47:00.086 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.123 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.044) 0:00:26.164 ***** 2020-11-19T02:47:00.130 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-19T02:47:00.131 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.167 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.043) 0:00:26.207 ***** 2020-11-19T02:47:00.174 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.211 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.044) 0:00:26.252 ***** 2020-11-19T02:47:00.218 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-19T02:47:00.219 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.271 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.059) 0:00:26.311 ***** 2020-11-19T02:47:00.279 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.316 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.044) 0:00:26.356 ***** 2020-11-19T02:47:00.322 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-19T02:47:00.323 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.360 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.044) 0:00:26.401 ***** 2020-11-19T02:47:00.367 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.404 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.043) 0:00:26.445 ***** 2020-11-19T02:47:00.410 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-19T02:47:00.412 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.447 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.042) 0:00:26.487 ***** 2020-11-19T02:47:00.454 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.493 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.046) 0:00:26.534 ***** 2020-11-19T02:47:00.500 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.536 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.042) 0:00:26.576 ***** 2020-11-19T02:47:00.543 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.580 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.043) 0:00:26.620 ***** 2020-11-19T02:47:00.586 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-19T02:47:00.588 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.626 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.047) 0:00:26.667 ***** 2020-11-19T02:47:00.633 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.670 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.043) 0:00:26.710 ***** 2020-11-19T02:47:00.677 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-11-19T02:47:00.679 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.713 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.043) 0:00:26.754 ***** 2020-11-19T02:47:00.720 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:00.760 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.046) 0:00:26.800 ***** 2020-11-19T02:47:00.766 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-19T02:47:00.768 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-19T02:47:00.809 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.049) 0:00:26.849 ***** 2020-11-19T02:47:00.817 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-19T02:47:00.860 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.050) 0:00:26.900 ***** 2020-11-19T02:47:00.866 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-19T02:47:00.868 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-19T02:47:00.906 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.046) 0:00:26.947 ***** 2020-11-19T02:47:00.913 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-19T02:47:00.952 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.045) 0:00:26.992 ***** 2020-11-19T02:47:00.958 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-19T02:47:00.960 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-19T02:47:01.002 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:00 +0000 (0:00:00.050) 0:00:27.043 ***** 2020-11-19T02:47:01.022 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-19T02:47:01.056 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.053) 0:00:27.096 ***** 2020-11-19T02:47:01.062 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi086.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-19T02:47:01.104 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.048) 0:00:27.145 ***** 2020-11-19T02:47:01.112 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-19T02:47:01.151 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.046) 0:00:27.191 ***** 2020-11-19T02:47:01.243 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-19T02:47:01.245 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-19T02:47:01.281 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.130) 0:00:27.322 ***** 2020-11-19T02:47:01.288 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-19T02:47:01.332 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.051) 0:00:27.373 ***** 2020-11-19T02:47:01.423 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-19T02:47:01.460 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.127) 0:00:27.500 ***** 2020-11-19T02:47:01.477 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-19T02:47:01.512 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.052) 0:00:27.553 ***** 2020-11-19T02:47:01.519 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-19T02:47:01.521 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:01.555 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.042) 0:00:27.596 ***** 2020-11-19T02:47:01.562 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:01.599 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.043) 0:00:27.639 ***** 2020-11-19T02:47:01.605 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-19T02:47:01.607 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:01.659 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.059) 0:00:27.699 ***** 2020-11-19T02:47:01.667 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:01.706 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.046) 0:00:27.746 ***** 2020-11-19T02:47:01.830 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-19T02:47:01.832 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-19T02:47:01.871 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.165) 0:00:27.911 ***** 2020-11-19T02:47:01.894 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-19T02:47:01.936 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:01 +0000 (0:00:00.064) 0:00:27.976 ***** 2020-11-19T02:47:02.397 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-19T02:47:02.399 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-19T02:47:02.434 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:02 +0000 (0:00:00.498) 0:00:28.475 ***** 2020-11-19T02:47:02.462 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-19T02:47:02.464 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi086"}, "changed": false} TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-19T02:47:02.504 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:02 +0000 (0:00:00.069) 0:00:28.544 ***** 2020-11-19T02:47:02.554 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:02 +0000 (0:00:00.050) 0:00:28.594 ***** 2020-11-19T02:47:02.605 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-19T02:47:02.607 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi160"}, "changed": false} 2020-11-19T02:47:02.644 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:02 +0000 (0:00:00.089) 0:00:28.684 ***** 2020-11-19T02:47:03.441 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi086.front.sepia.ceph.com", "ansible_hostname": "smithi086", "ansible_nodename": "smithi086"}, "changed": true, "name": "smithi086"} 2020-11-19T02:47:03.479 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:03 +0000 (0:00:00.835) 0:00:29.519 ***** 2020-11-19T02:47:03.486 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi160.front.sepia.ceph.com", "ansible_hostname": "smithi160", "ansible_nodename": "smithi160"}, "changed": true, "name": "smithi160"} 2020-11-19T02:47:03.525 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:03 +0000 (0:00:00.044) 0:00:29.564 ***** 2020-11-19T02:47:03.814 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-11-19T02:47:03.816 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-19T02:47:03.854 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:03 +0000 (0:00:00.330) 0:00:29.894 ***** 2020-11-19T02:47:03.861 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-19T02:47:03.901 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:03 +0000 (0:00:00.046) 0:00:29.941 ***** 2020-11-19T02:47:04.205 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-11-19T02:47:04.207 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-19T02:47:04.243 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:04 +0000 (0:00:00.342) 0:00:30.284 ***** 2020-11-19T02:47:04.253 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-19T02:47:04.287 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:04 +0000 (0:00:00.044) 0:00:30.328 ***** 2020-11-19T02:47:04.742 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-19T02:47:04.744 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-19T02:47:04.780 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:04 +0000 (0:00:00.493) 0:00:30.821 ***** 2020-11-19T02:47:04.789 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-19T02:47:04.830 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:04 +0000 (0:00:00.049) 0:00:30.870 ***** 2020-11-19T02:47:04.837 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-19T02:47:04.839 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:04.873 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:04 +0000 (0:00:00.043) 0:00:30.913 ***** 2020-11-19T02:47:04.880 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:04.917 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:04 +0000 (0:00:00.043) 0:00:30.957 ***** 2020-11-19T02:47:04.924 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:04.962 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:04 +0000 (0:00:00.045) 0:00:31.003 ***** 2020-11-19T02:47:04.969 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.007 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.045) 0:00:31.048 ***** 2020-11-19T02:47:05.014 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-19T02:47:05.016 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.050 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.042) 0:00:31.091 ***** 2020-11-19T02:47:05.057 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.093 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.043) 0:00:31.134 ***** 2020-11-19T02:47:05.102 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.138 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.042) 0:00:31.177 ***** 2020-11-19T02:47:05.143 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.179 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.042) 0:00:31.220 ***** 2020-11-19T02:47:05.186 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-19T02:47:05.188 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.223 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.043) 0:00:31.263 ***** 2020-11-19T02:47:05.229 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.268 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.045) 0:00:31.309 ***** 2020-11-19T02:47:05.274 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-19T02:47:05.276 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.310 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.042) 0:00:31.351 ***** 2020-11-19T02:47:05.317 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.353 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.042) 0:00:31.394 ***** 2020-11-19T02:47:05.361 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-19T02:47:05.363 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.399 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.045) 0:00:31.439 ***** 2020-11-19T02:47:05.407 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.456 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.057) 0:00:31.497 ***** 2020-11-19T02:47:05.462 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-11-19T02:47:05.464 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.502 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.043) 0:00:31.541 ***** 2020-11-19T02:47:05.633 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.666 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.165) 0:00:31.706 ***** 2020-11-19T02:47:05.673 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-19T02:47:05.708 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.042) 0:00:31.749 ***** 2020-11-19T02:47:05.754 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.045) 0:00:31.794 ***** 2020-11-19T02:47:05.764 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-19T02:47:05.806 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.052) 0:00:31.847 ***** 2020-11-19T02:47:05.851 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.044) 0:00:31.891 ***** 2020-11-19T02:47:05.857 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-19T02:47:05.896 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.045) 0:00:31.936 ***** 2020-11-19T02:47:05.940 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.043) 0:00:31.980 ***** 2020-11-19T02:47:05.947 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:05.985 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:05 +0000 (0:00:00.043) 0:00:32.024 ***** 2020-11-19T02:47:05.990 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.028 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.044) 0:00:32.069 ***** 2020-11-19T02:47:06.034 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-11-19T02:47:06.036 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.074 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.045) 0:00:32.114 ***** 2020-11-19T02:47:06.080 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.082 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.117 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.043) 0:00:32.158 ***** 2020-11-19T02:47:06.124 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-19T02:47:06.126 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.161 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.043) 0:00:32.202 ***** 2020-11-19T02:47:06.168 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.205 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.043) 0:00:32.245 ***** 2020-11-19T02:47:06.211 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:47:06.213 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.260 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.054) 0:00:32.300 ***** 2020-11-19T02:47:06.268 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.307 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.047) 0:00:32.348 ***** 2020-11-19T02:47:06.315 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-19T02:47:06.317 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.352 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.045) 0:00:32.393 ***** 2020-11-19T02:47:06.359 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.399 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.046) 0:00:32.440 ***** 2020-11-19T02:47:06.406 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-19T02:47:06.407 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.443 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.043) 0:00:32.483 ***** 2020-11-19T02:47:06.449 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.487 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.044) 0:00:32.528 ***** 2020-11-19T02:47:06.494 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-19T02:47:06.495 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.530 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.042) 0:00:32.571 ***** 2020-11-19T02:47:06.537 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.573 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.042) 0:00:32.614 ***** 2020-11-19T02:47:06.580 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-19T02:47:06.582 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.617 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.043) 0:00:32.657 ***** 2020-11-19T02:47:06.623 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.676 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.059) 0:00:32.717 ***** 2020-11-19T02:47:06.682 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-19T02:47:06.684 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.720 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.043) 0:00:32.761 ***** 2020-11-19T02:47:06.727 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.767 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.047) 0:00:32.808 ***** 2020-11-19T02:47:06.776 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-19T02:47:06.778 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.825 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.057) 0:00:32.865 ***** 2020-11-19T02:47:06.834 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.886 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.057) 0:00:32.923 ***** 2020-11-19T02:47:06.890 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-19T02:47:06.891 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.936 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.054) 0:00:32.977 ***** 2020-11-19T02:47:06.953 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:06.990 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:06 +0000 (0:00:00.053) 0:00:33.030 ***** 2020-11-19T02:47:06.998 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.034 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.044) 0:00:33.075 ***** 2020-11-19T02:47:07.042 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.083 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.048) 0:00:33.123 ***** 2020-11-19T02:47:07.089 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-19T02:47:07.091 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.128 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.045) 0:00:33.169 ***** 2020-11-19T02:47:07.135 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.172 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.043) 0:00:33.213 ***** 2020-11-19T02:47:07.178 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-19T02:47:07.180 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.217 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.044) 0:00:33.258 ***** 2020-11-19T02:47:07.226 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.262 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.045) 0:00:33.303 ***** 2020-11-19T02:47:07.269 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-19T02:47:07.271 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.333 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.070) 0:00:33.373 ***** 2020-11-19T02:47:07.348 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.398 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.060) 0:00:33.434 ***** 2020-11-19T02:47:07.561 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 02:47:07 +0000 (0:00:00.044) 0:00:33.479 ***** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 02:47:07 +0000 (0:00:00.047) 0:00:33.526 ***** TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 02:47:07 +0000 (0:00:00.049) 0:00:33.575 ***** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.579 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.044) 0:00:33.620 ***** 2020-11-19T02:47:07.585 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-19T02:47:07.587 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.627 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.047) 0:00:33.667 ***** 2020-11-19T02:47:07.633 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.675 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.046) 0:00:33.714 ***** 2020-11-19T02:47:07.680 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-19T02:47:07.681 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.716 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.043) 0:00:33.757 ***** 2020-11-19T02:47:07.723 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.760 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.043) 0:00:33.801 ***** 2020-11-19T02:47:07.766 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-19T02:47:07.768 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.804 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.043) 0:00:33.845 ***** 2020-11-19T02:47:07.811 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.848 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.043) 0:00:33.888 ***** 2020-11-19T02:47:07.854 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-19T02:47:07.856 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.891 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.042) 0:00:33.931 ***** 2020-11-19T02:47:07.897 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.937 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.046) 0:00:33.977 ***** 2020-11-19T02:47:07.942 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-19T02:47:07.944 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:07.982 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:07 +0000 (0:00:00.044) 0:00:34.022 ***** 2020-11-19T02:47:07.990 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:08.029 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:08 +0000 (0:00:00.047) 0:00:34.069 ***** 2020-11-19T02:47:08.107 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-19T02:47:08.109 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005196", "end": "2020-11-19 02:47:08.222736", "rc": 0, "start": "2020-11-19 02:47:08.217540", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:08.185 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:08 +0000 (0:00:00.156) 0:00:34.226 ***** 2020-11-19T02:47:08.194 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004502", "end": "2020-11-19 02:47:07.895671", "rc": 0, "start": "2020-11-19 02:47:07.891169", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:08.238 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:08 +0000 (0:00:00.053) 0:00:34.279 ***** 2020-11-19T02:47:08.245 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-19T02:47:08.296 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:08 +0000 (0:00:00.057) 0:00:34.336 ***** 2020-11-19T02:47:08.342 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:08 +0000 (0:00:00.045) 0:00:34.382 ***** 2020-11-19T02:47:08.642 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-19T02:47:08.645 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-19T02:47:08.691 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:08 +0000 (0:00:00.349) 0:00:34.732 ***** 2020-11-19T02:47:08.700 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-19T02:47:08.742 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:08 +0000 (0:00:00.050) 0:00:34.782 ***** 2020-11-19T02:47:08.750 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-19T02:47:08.751 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:08.847 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:08 +0000 (0:00:00.102) 0:00:34.885 ***** 2020-11-19T02:47:08.864 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:08.917 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:08 +0000 (0:00:00.071) 0:00:34.957 ***** 2020-11-19T02:47:10.038 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi086.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-19T02:47:10.060 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-19T02:47:10.062 INFO:teuthology.task.ansible.out: 2020-11-19T02:47:10.993 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-19T02:47:11.052 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:11 +0000 (0:00:02.135) 0:00:37.092 ***** 2020-11-19T02:47:11.061 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-19T02:47:11.100 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:11 +0000 (0:00:00.048) 0:00:37.141 ***** 2020-11-19T02:47:11.995 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-19T02:47:11.997 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:47:12.031 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:12 +0000 (0:00:00.931) 0:00:38.072 ***** 2020-11-19T02:47:12.041 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:47:12.078 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:12 +0000 (0:00:00.046) 0:00:38.118 ***** 2020-11-19T02:47:12.086 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-19T02:47:12.123 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:12 +0000 (0:00:00.045) 0:00:38.164 ***** 2020-11-19T02:47:12.169 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:12 +0000 (0:00:00.045) 0:00:38.210 ***** 2020-11-19T02:47:12.285 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo 2020-11-19T02:47:12.287 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.033628", "end": "2020-11-19 02:47:12.394626", "rc": 0, "start": "2020-11-19 02:47:12.360998", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:12.335 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:12 +0000 (0:00:00.165) 0:00:38.376 ***** 2020-11-19T02:47:12.342 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034366", "end": "2020-11-19 02:47:12.068763", "rc": 0, "start": "2020-11-19 02:47:12.034397", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:12.397 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:12 +0000 (0:00:00.062) 0:00:38.438 ***** 2020-11-19T02:47:15.106 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-19T02:47:15.108 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:47:15.143 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:15 +0000 (0:00:02.745) 0:00:41.183 ***** 2020-11-19T02:47:15.330 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:47:15.367 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:15 +0000 (0:00:00.224) 0:00:41.408 ***** 2020-11-19T02:47:23.540 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-11-19T02:47:23.541 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 13 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137606 files and directorie 2020-11-19T02:47:23.543 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 13 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-11-19T02:47:23.545 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137606 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-19T02:47:23.604 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:23 +0000 (0:00:08.236) 0:00:49.644 ***** 2020-11-19T02:47:24.407 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 13 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137606 files and directorie 2020-11-19T02:47:24.410 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 13 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-11-19T02:47:24.413 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137606 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-19T02:47:24.446 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:24 +0000 (0:00:00.839) 0:00:50.484 ***** 2020-11-19T02:47:24.462 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:47:24.503 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:24 +0000 (0:00:00.057) 0:00:50.542 ***** 2020-11-19T02:47:24.530 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:24.566 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:24 +0000 (0:00:00.064) 0:00:50.606 ***** 2020-11-19T02:47:25.355 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-19T02:47:25.357 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:47:25.392 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:25 +0000 (0:00:00.825) 0:00:51.432 ***** 2020-11-19T02:47:25.420 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-19T02:47:25.422 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:25.456 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:25 +0000 (0:00:00.064) 0:00:51.497 ***** 2020-11-19T02:47:33.387 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-19T02:47:33.388 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9879 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-11-19T02:47:33.390 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137464 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-11-19T02:47:33.391 INFO:teuthology.task.ansible.out:u1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9879 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-11-19T02:47:33.393 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137464 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-19T02:47:33.438 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:33 +0000 (0:00:07.982) 0:00:59.479 ***** 2020-11-19T02:47:34.270 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:47:34.330 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:34 +0000 (0:00:00.888) 0:01:00.367 ***** 2020-11-19T02:47:34.478 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-19T02:47:34.480 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9593 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-11-19T02:47:34.482 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137464 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-11-19T02:47:34.484 INFO:teuthology.task.ansible.out:u1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9593 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-11-19T02:47:34.486 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137464 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-19T02:47:34.534 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:34 +0000 (0:00:00.204) 0:01:00.572 ***** 2020-11-19T02:47:34.539 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-19T02:47:34.540 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-19T02:47:34.581 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:34 +0000 (0:00:00.048) 0:01:00.621 ***** 2020-11-19T02:47:34.625 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-19T02:47:34.628 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:34.675 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:34 +0000 (0:00:00.094) 0:01:00.716 ***** 2020-11-19T02:47:34.820 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-19T02:47:34.822 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-19T02:47:34.855 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:34 +0000 (0:00:00.180) 0:01:00.896 ***** 2020-11-19T02:47:35.238 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-19T02:47:35.240 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-19T02:47:35.275 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:35 +0000 (0:00:00.419) 0:01:01.316 ***** 2020-11-19T02:47:35.411 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-19T02:47:35.414 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:47:35.451 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:35 +0000 (0:00:00.175) 0:01:01.491 ***** 2020-11-19T02:47:35.502 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-19T02:47:35.505 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-19T02:47:35.562 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:35 +0000 (0:00:00.110) 0:01:01.602 ***** 2020-11-19T02:47:35.594 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-19T02:47:35.647 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:35 +0000 (0:00:00.085) 0:01:01.687 ***** 2020-11-19T02:47:35.678 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-19T02:47:35.680 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:35.737 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:35 +0000 (0:00:00.089) 0:01:01.777 ***** 2020-11-19T02:47:35.743 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-19T02:47:35.747 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T02:47:35.782 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:35 +0000 (0:00:00.045) 0:01:01.822 ***** 2020-11-19T02:47:35.890 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-19T02:47:35.892 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-19T02:47:35.939 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:35 +0000 (0:00:00.154) 0:01:01.977 ***** 2020-11-19T02:47:36.105 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-19T02:47:36.107 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 02:46:11 UTC", "ActiveEnterTimestampMonotonic": "20832413", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service basic.target system.slice systemd-journald.socket -.mount remote-fs.target network.target nss-lookup.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 02:46:11 UTC", "AssertTimestampMonotonic": "20400078", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 02:46:11 UTC", "ConditionTimestampMonotonic": "20400078", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-19T02:47:36.109 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1216", "ExecMainStartTimestamp": "Thu 2020-11-19 02:46:11 UTC", "ExecMainStartTimestampMonotonic": "20832384", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 02:46:11 UTC", "InactiveExitTimestampMonotonic": "20401546", "InvocationID": "495258a07036466c81bba88ab16aa282", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-19T02:47:36.112 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1216", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-19T02:47:36.114 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 02:46:11 UTC", "StateChangeTimestampMonotonic": "20832413", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 02:46:11 UTC", "WatchdogTimestampMonotonic": "20832411", "WatchdogUSec": "0"}} 2020-11-19T02:47:36.147 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.210) 0:01:02.188 ***** 2020-11-19T02:47:36.175 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-19T02:47:36.177 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.224 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.076) 0:01:02.265 ***** 2020-11-19T02:47:36.254 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.292 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.067) 0:01:02.332 ***** 2020-11-19T02:47:36.320 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-19T02:47:36.322 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.368 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.075) 0:01:02.408 ***** 2020-11-19T02:47:36.405 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.037) 0:01:02.446 ***** 2020-11-19T02:47:36.412 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-19T02:47:36.414 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.455 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.049) 0:01:02.496 ***** 2020-11-19T02:47:36.486 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.521 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.066) 0:01:02.562 ***** 2020-11-19T02:47:36.559 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-19T02:47:36.561 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi160.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-19T02:47:36.596 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.074) 0:01:02.636 ***** 2020-11-19T02:47:36.630 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.033) 0:01:02.670 ***** 2020-11-19T02:47:36.635 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-19T02:47:36.637 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.676 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.046) 0:01:02.717 ***** 2020-11-19T02:47:36.704 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-19T02:47:36.706 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.750 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.073) 0:01:02.790 ***** 2020-11-19T02:47:36.770 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-19T02:47:36.772 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T02:47:36.808 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.057) 0:01:02.848 ***** 2020-11-19T02:47:36.815 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.855 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.046) 0:01:02.895 ***** 2020-11-19T02:47:36.883 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.931 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.076) 0:01:02.971 ***** 2020-11-19T02:47:36.959 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:36.994 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:36 +0000 (0:00:00.062) 0:01:03.034 ***** 2020-11-19T02:47:37.024 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.094 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.093) 0:01:03.128 ***** 2020-11-19T02:47:37.108 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 02:46:23 UTC", "ActiveEnterTimestampMonotonic": "21112593", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target -.mount basic.target nss-lookup.target sysinit.target systemd-journald.socket system.slice systemd-tmpfiles-setup.service remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 02:46:23 UTC", "AssertTimestampMonotonic": "20656058", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 02:46:23 UTC", "ConditionTimestampMonotonic": "20656051", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", 2020-11-19T02:47:37.110 INFO:teuthology.task.ansible.out:"Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1347", "ExecMainStartTimestamp": "Thu 2020-11-19 02:46:23 UTC", "ExecMainStartTimestampMonotonic": "21112580", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 02:46:23 UTC", "InactiveExitTimestampMonotonic": "20658577", "InvocationID": "fc36c429c90a419d87560f6a87546746", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA 2020-11-19T02:47:37.111 INFO:teuthology.task.ansible.out:": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1347", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", 2020-11-19T02:47:37.113 INFO:teuthology.task.ansible.out: "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 02:46:23 UTC", "StateChangeTimestampMonotonic": "21112593", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 02:46:23 UTC", "WatchdogTimestampMonotonic": "21112592", "WatchdogUSec": "0"}} 2020-11-19T02:47:37.151 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.063) 0:01:03.191 ***** 2020-11-19T02:47:37.158 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-19T02:47:37.160 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.204 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.053) 0:01:03.245 ***** 2020-11-19T02:47:37.213 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-19T02:47:37.214 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.253 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.048) 0:01:03.293 ***** 2020-11-19T02:47:37.259 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-19T02:47:37.262 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.305 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.052) 0:01:03.346 ***** 2020-11-19T02:47:37.313 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.350 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.044) 0:01:03.390 ***** 2020-11-19T02:47:37.356 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-19T02:47:37.358 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.393 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.043) 0:01:03.433 ***** 2020-11-19T02:47:37.399 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-19T02:47:37.401 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.441 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.047) 0:01:03.481 ***** 2020-11-19T02:47:37.449 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-19T02:47:37.450 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.488 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.048) 0:01:03.529 ***** 2020-11-19T02:47:37.496 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-19T02:47:37.498 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.532 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.043) 0:01:03.572 ***** 2020-11-19T02:47:37.538 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-19T02:47:37.540 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.578 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.044) 0:01:03.617 ***** 2020-11-19T02:47:37.583 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-19T02:47:37.585 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.633 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.054) 0:01:03.671 ***** 2020-11-19T02:47:37.639 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.681 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.049) 0:01:03.721 ***** 2020-11-19T02:47:37.687 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-19T02:47:37.689 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.726 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.045) 0:01:03.766 ***** 2020-11-19T02:47:37.732 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-19T02:47:37.734 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.774 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.046) 0:01:03.812 ***** 2020-11-19T02:47:37.784 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-19T02:47:37.787 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.836 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.064) 0:01:03.877 ***** 2020-11-19T02:47:37.846 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-19T02:47:37.848 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.884 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.047) 0:01:03.924 ***** 2020-11-19T02:47:37.890 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-19T02:47:37.892 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.927 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.043) 0:01:03.968 ***** 2020-11-19T02:47:37.934 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:37.972 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:37 +0000 (0:00:00.044) 0:01:04.013 ***** 2020-11-19T02:47:37.979 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:47:37.981 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.016 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.044) 0:01:04.057 ***** 2020-11-19T02:47:38.023 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.061 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.042) 0:01:04.100 ***** 2020-11-19T02:47:38.067 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-19T02:47:38.069 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.103 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.043) 0:01:04.143 ***** 2020-11-19T02:47:38.109 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.153 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.050) 0:01:04.194 ***** 2020-11-19T02:47:38.162 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.205 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.051) 0:01:04.245 ***** 2020-11-19T02:47:38.211 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-19T02:47:38.212 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.254 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.049) 0:01:04.294 ***** 2020-11-19T02:47:38.261 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-19T02:47:38.263 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.302 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.048) 0:01:04.343 ***** 2020-11-19T02:47:38.309 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-19T02:47:38.310 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.348 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.046) 0:01:04.389 ***** 2020-11-19T02:47:38.355 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-19T02:47:38.356 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.402 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.053) 0:01:04.442 ***** 2020-11-19T02:47:38.415 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-19T02:47:38.417 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.453 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.051) 0:01:04.493 ***** 2020-11-19T02:47:38.465 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.499 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.045) 0:01:04.539 ***** 2020-11-19T02:47:38.528 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.564 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.063) 0:01:04.603 ***** 2020-11-19T02:47:38.591 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-19T02:47:38.593 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.645 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.081) 0:01:04.684 ***** 2020-11-19T02:47:38.664 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-19T02:47:38.666 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-19T02:47:38.717 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.073) 0:01:04.757 ***** 2020-11-19T02:47:38.725 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-19T02:47:38.726 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.781 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.063) 0:01:04.821 ***** 2020-11-19T02:47:38.818 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-19T02:47:38.820 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.853 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.073) 0:01:04.894 ***** 2020-11-19T02:47:38.881 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-19T02:47:38.883 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.917 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.062) 0:01:04.957 ***** 2020-11-19T02:47:38.950 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:38.985 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:38 +0000 (0:00:00.069) 0:01:05.026 ***** 2020-11-19T02:47:39.014 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:39.049 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.063) 0:01:05.089 ***** 2020-11-19T02:47:39.076 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-19T02:47:39.078 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-19T02:47:39.113 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.064) 0:01:05.153 ***** 2020-11-19T02:47:39.147 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.033) 0:01:05.187 ***** 2020-11-19T02:47:39.154 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-19T02:47:39.156 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:39.191 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.044) 0:01:05.232 ***** 2020-11-19T02:47:39.199 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:39.236 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.044) 0:01:05.277 ***** 2020-11-19T02:47:39.286 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:39.322 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.085) 0:01:05.362 ***** 2020-11-19T02:47:39.352 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:39.405 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.083) 0:01:05.445 ***** 2020-11-19T02:47:39.433 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-19T02:47:39.435 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:39.480 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.075) 0:01:05.520 ***** 2020-11-19T02:47:39.606 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-19T02:47:39.616 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-19T02:47:39.646 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.166) 0:01:05.687 ***** 2020-11-19T02:47:39.653 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-19T02:47:39.655 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-19T02:47:39.710 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:39 +0000 (0:00:00.063) 0:01:05.750 ***** 2020-11-19T02:47:39.983 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-11-19T02:47:39.985 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-19T02:47:40.030 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.319) 0:01:06.070 ***** 2020-11-19T02:47:40.042 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-19T02:47:40.093 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.061) 0:01:06.131 ***** 2020-11-19T02:47:40.120 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:40.165 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.072) 0:01:06.203 ***** 2020-11-19T02:47:40.441 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-19T02:47:40.444 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-19T02:47:40.483 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.320) 0:01:06.524 ***** 2020-11-19T02:47:40.514 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-19T02:47:40.516 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-19T02:47:40.553 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.069) 0:01:06.594 ***** 2020-11-19T02:47:40.644 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-19T02:47:40.646 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T02:47:40.682 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.129) 0:01:06.723 ***** 2020-11-19T02:47:40.709 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-19T02:47:40.711 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:40.745 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.062) 0:01:06.786 ***** 2020-11-19T02:47:40.773 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:40.808 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.063) 0:01:06.849 ***** 2020-11-19T02:47:40.846 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-19T02:47:40.848 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:40.883 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.073) 0:01:06.922 ***** 2020-11-19T02:47:40.911 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:40.960 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:40 +0000 (0:00:00.077) 0:01:07.000 ***** 2020-11-19T02:47:40.967 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-11-19T02:47:40.969 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-19T02:47:41.023 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.063) 0:01:07.063 ***** 2020-11-19T02:47:41.031 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-19T02:47:41.033 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.084 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.060) 0:01:07.124 ***** 2020-11-19T02:47:41.116 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:47:41.118 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-19T02:47:41.151 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.067) 0:01:07.192 ***** 2020-11-19T02:47:41.220 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi086.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.229 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.231 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.241 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.243 INFO:teuthology.task.ansible.out: 2020-11-19T02:47:41.253 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.265 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.266 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-19T02:47:41.278 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.290 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.301 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.313 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.324 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.337 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.382 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.230) 0:01:07.422 ***** 2020-11-19T02:47:41.389 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-19T02:47:41.390 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-19T02:47:41.428 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.045) 0:01:07.468 ***** 2020-11-19T02:47:41.555 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi160.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T02:47:41.596 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.168) 0:01:07.637 ***** 2020-11-19T02:47:41.603 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-19T02:47:41.606 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003825", "end": "2020-11-19 02:47:41.565619", "rc": 0, "start": "2020-11-19 02:47:41.561794", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-19T02:47:41.660 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.063) 0:01:07.700 ***** 2020-11-19T02:47:41.666 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-19T02:47:41.668 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.704 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.044) 0:01:07.745 ***** 2020-11-19T02:47:41.713 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-19T02:47:41.715 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.761 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.056) 0:01:07.801 ***** 2020-11-19T02:47:41.768 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.804 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.043) 0:01:07.845 ***** 2020-11-19T02:47:41.812 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.848 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.043) 0:01:07.889 ***** 2020-11-19T02:47:41.855 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-19T02:47:41.857 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.895 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.046) 0:01:07.935 ***** 2020-11-19T02:47:41.901 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-19T02:47:41.902 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.938 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.043) 0:01:07.979 ***** 2020-11-19T02:47:41.945 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-19T02:47:41.946 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:41.982 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:41 +0000 (0:00:00.044) 0:01:08.023 ***** 2020-11-19T02:47:42.020 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-19T02:47:42.022 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.064 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:42 +0000 (0:00:00.080) 0:01:08.104 ***** 2020-11-19T02:47:42.125 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:47:42.127 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-19T02:47:42.161 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:42 +0000 (0:00:00.098) 0:01:08.202 ***** 2020-11-19T02:47:42.220 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-19T02:47:42.222 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.232 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.244 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.246 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.256 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.270 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-19T02:47:42.279 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.281 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.293 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.304 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.306 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.316 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.327 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.330 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.370 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:42 +0000 (0:00:00.208) 0:01:08.410 ***** 2020-11-19T02:47:42.577 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-19T02:47:42.579 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003866", "end": "2020-11-19 02:47:42.540454", "rc": 0, "start": "2020-11-19 02:47:42.536588", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-19T02:47:42.627 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:42 +0000 (0:00:00.257) 0:01:08.667 ***** 2020-11-19T02:47:42.668 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-19T02:47:42.670 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.720 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:42 +0000 (0:00:00.093) 0:01:08.761 ***** 2020-11-19T02:47:42.762 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.801 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:42 +0000 (0:00:00.080) 0:01:08.841 ***** 2020-11-19T02:47:42.830 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:47:42.832 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:42.892 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:42 +0000 (0:00:00.090) 0:01:08.932 ***** 2020-11-19T02:47:42.925 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:42 +0000 (0:00:00.033) 0:01:08.966 ***** 2020-11-19T02:47:43.035 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 02:47:42 +0000 (0:00:00.044) 0:01:09.010 ***** TASK [testnode : Set root disk] ************************************************ ok: [smithi086.front.sepia.ceph.com] => (item={'block_used': 13753454, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341654, 'size_available': 927095414784, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341654, "block_size": 4096, "block_total": 240095108, "block_used": 13753454, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095414784, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}} 2020-11-19T02:47:43.048 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:43 +0000 (0:00:00.077) 0:01:09.088 ***** 2020-11-19T02:47:43.214 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009842", "end": "2020-11-19 02:47:43.307292", "rc": 0, "start": "2020-11-19 02:47:43.297450", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:43.254 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:43 +0000 (0:00:00.206) 0:01:09.295 ***** 2020-11-19T02:47:43.292 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-19T02:47:43.294 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'block_used': 13753454, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341654, 'size_available': 927095414784, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341654, "block_size": 4096, "block_total": 240095108, "block_used": 13753454, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095414784, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:43.329 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:43 +0000 (0:00:00.074) 0:01:09.369 ***** 2020-11-19T02:47:43.358 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-19T02:47:43.391 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:43 +0000 (0:00:00.062) 0:01:09.432 ***** 2020-11-19T02:47:43.440 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-19T02:47:43.442 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:43.825 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:47:43.859 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:43 +0000 (0:00:00.467) 0:01:09.899 ***** 2020-11-19T02:47:43.888 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:43.931 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:43 +0000 (0:00:00.071) 0:01:09.971 ***** 2020-11-19T02:47:43.967 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-19T02:47:43.969 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item={'block_used': 13753457, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341651, 'size_available': 927095402496, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341651, "block_size": 4096, "block_total": 240095108, "block_used": 13753457, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095402496, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}} 2020-11-19T02:47:44.004 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:43 +0000 (0:00:00.074) 0:01:10.045 ***** 2020-11-19T02:47:44.136 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-19T02:47:44.138 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009659", "end": "2020-11-19 02:47:43.881834", "rc": 0, "start": "2020-11-19 02:47:43.872175", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:44.188 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:44 +0000 (0:00:00.183) 0:01:10.228 ***** 2020-11-19T02:47:44.227 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-19T02:47:44.229 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'block_used': 13753457, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341651, 'size_available': 927095402496, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341651, "block_size": 4096, "block_total": 240095108, "block_used": 13753457, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095402496, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:44.280 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:44 +0000 (0:00:00.092) 0:01:10.321 ***** 2020-11-19T02:47:44.309 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-19T02:47:44.345 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:44 +0000 (0:00:00.064) 0:01:10.385 ***** 2020-11-19T02:47:44.404 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-19T02:47:44.407 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:45.442 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYcVhrK6ROwhyJEJcstPtZmFiaKdW3R63U'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.888337", "end": "2020-11-19 02:47:45.532219", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYcVhrK6ROwhyJEJcstPtZmFiaKdW3R63U"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:43.643882", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:45.464 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:45.488 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:46.494 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MKapE0CsIfWQuqdmtZsfd7daOflNOJUAo'], 'uuids': ['3312f5d4-6c50-4f49-baac-14cbe37be913']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.962746", "end": "2020-11-19 02:47:46.230642", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MKapE0CsIfWQuqdmtZsfd7daOflNOJUAo"], "labels": [], "masters": [], "uuids": ["3312f5d4-6c50-4f49-baac-14cbe37be913"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:44.267896", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:46.506 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:46.517 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:46.749 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYU3u1ktiXUW2SsWi0zt4aD9qHcKKZmPDR'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.138601", "end": "2020-11-19 02:47:46.835091", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYU3u1ktiXUW2SsWi0zt4aD9qHcKKZmPDR"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:45.696490", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:46.761 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGQ8', 'wwn-0x5000c500918e8385'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGQ8-part1', 'wwn-0x5000c500918e8385-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e8385', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LGQ8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGQ8", "wwn-0x5000c500918e8385"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGQ8-part1", "wwn-0x5000c500918e8385-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LGQ8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e8385"}}, "skip_reason": 2020-11-19T02:47:46.763 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-19T02:47:47.782 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3McSr5kqzzAbN62mS4i0tNjjsdlEUsRNpZ'], 'uuids': ['7fc7a097-9dd3-466d-be50-06e50161c171']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.142824", "end": "2020-11-19 02:47:47.518779", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3McSr5kqzzAbN62mS4i0tNjjsdlEUsRNpZ"], "labels": [], "masters": [], "uuids": ["7fc7a097-9dd3-466d-be50-06e50161c171"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:46.375955", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:47.783 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFZB', 'wwn-0x5000c5009292466b'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFZB-part1', 'wwn-0x5000c5009292466b-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009292466b', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFZB', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFZB", "wwn-0x5000c5009292466b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFZB-part1", "wwn-0x5000c5009292466b-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFZB", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292466b"}}, "skip_reason": 2020-11-19T02:47:47.785 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-19T02:47:48.049 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-uHQcaA-oe3Z-cnoC-FUmy-CKas-6y2j-4ksMlD', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400UL400BGN', 'nvme-nvme.8086-50484654363230343030554c34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.169085", "end": "2020-11-19 02:47:48.128336", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-uHQcaA-oe3Z-cnoC-FUmy-CKas-6y2j-4ksMlD", "nvme-INTEL_SSDPEDMD400G4_PHFT620400UL400BGN", "nvme-nvme.8086-50484654363230343030554c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:46.959251", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-19T02:47:48.330 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_re 2020-11-19T02:47:48.332 INFO:teuthology.task.ansible.out:ason": "Conditional result was False"} 2020-11-19T02:47:49.057 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-0GNJMp-Ir33-1IFl-74cc-ys8I-dFk4-KflDjO', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300JY400BGN', 'nvme-nvme.8086-435646543632333330304a5934303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.144316", "end": "2020-11-19 02:47:48.792839", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-0GNJMp-Ir33-1IFl-74cc-ys8I-dFk4-KflDjO", "nvme-INTEL_SSDPEDMD400G4_CVFT623300JY400BGN", "nvme-nvme.8086-435646543632333330304a5934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:47.648523", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-19T02:47:49.058 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:49.074 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:49.360 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYhJWrxQ5m7CqTHPhgzieL3qKuq90TOT4X'], 'uuids': ['830a44af-55bd-4af3-93f6-8e82e8e3b8ea']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.161213", "end": "2020-11-19 02:47:49.443246", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYhJWrxQ5m7CqTHPhgzieL3qKuq90TOT4X"], "labels": [], "masters": [], "uuids": ["830a44af-55bd-4af3-93f6-8e82e8e3b8ea"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:48.282033", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:49.372 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:49.373 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:50.381 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MQcKSk194GlrtHrJWkiZyI6QWX2aM6CKY'], 'uuids': ['600855cd-7ba5-48c4-a8bf-40fb7bea3afa']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.181539", "end": "2020-11-19 02:47:50.116074", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MQcKSk194GlrtHrJWkiZyI6QWX2aM6CKY"], "labels": [], "masters": [], "uuids": ["600855cd-7ba5-48c4-a8bf-40fb7bea3afa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:48.934535", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:50.383 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:50.395 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:50.397 INFO:teuthology.task.ansible.out: 2020-11-19T02:47:50.645 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYupgYo0wWdYRsa2v0ivQjfpc6nsRa9M6D'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.142420", "end": "2020-11-19 02:47:50.726161", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYupgYo0wWdYRsa2v0ivQjfpc6nsRa9M6D"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:49.583741", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:50.667 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:51.672 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MtMx1s8E8BDDxpYOzypawpwZ70WCfLu81'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.125154", "end": "2020-11-19 02:47:51.412439", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MtMx1s8E8BDDxpYOzypawpwZ70WCfLu81"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:50.287285", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:51.681 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:51.903 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYNhCEwiEGRj6rMOWhng19Wtw97vmriIY2'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.121130", "end": "2020-11-19 02:47:51.984211", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYNhCEwiEGRj6rMOWhng19Wtw97vmriIY2"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:50.863081", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:51.949 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:51 +0000 (0:00:07.604) 0:01:17.989 ***** 2020-11-19T02:47:51.998 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-19T02:47:52.000 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:52.286 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYcVhrK6ROwhyJEJcstPtZmFiaKdW3R63U'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.158666", "end": "2020-11-19 02:47:52.356143", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYcVhrK6ROwhyJEJcstPtZmFiaKdW3R63U"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:52.197477", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154358 s, 747 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154358 s, 747 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:52.288 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:52.309 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:52.671 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYU3u1ktiXUW2SsWi0zt4aD9qHcKKZmPDR'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.152798", "end": "2020-11-19 02:47:52.749609", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYU3u1ktiXUW2SsWi0zt4aD9qHcKKZmPDR"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:52.596811", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148353 s, 777 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148353 s, 777 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:52.686 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGQ8', 'wwn-0x5000c500918e8385'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGQ8-part1', 'wwn-0x5000c500918e8385-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e8385', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LGQ8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGQ8", "wwn-0x5000c500918e8385"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGQ8-part1", "wwn-0x5000c500918e8385-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LGQ8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e8385"}}, "skip_reason": 2020-11-19T02:47:52.688 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-19T02:47:52.867 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-uHQcaA-oe3Z-cnoC-FUmy-CKas-6y2j-4ksMlD', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400UL400BGN', 'nvme-nvme.8086-50484654363230343030554c34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.050961", "end": "2020-11-19 02:47:52.944836", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-uHQcaA-oe3Z-cnoC-FUmy-CKas-6y2j-4ksMlD", "nvme-INTEL_SSDPEDMD400G4_PHFT620400UL400BGN", "nvme-nvme.8086-50484654363230343030554c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:52.893875", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0463157 s, 2.5 GB/s", "std 2020-11-19T02:47:52.869 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0463157 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:52.879 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:52.880 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:52.903 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-19T02:47:52.905 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MHti4zBRFjircGffvVPJi7RSUGt3ftmGB'], 'uuids': ['2877b0c1-c746-4d42-af1a-c50e1e92f037']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.106644", "end": "2020-11-19 02:47:52.646827", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MHti4zBRFjircGffvVPJi7RSUGt3ftmGB"], "labels": [], "masters": [], "uuids": ["2877b0c1-c746-4d42-af1a-c50e1e92f037"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:51.540183", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T02:47:52.975 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:52 +0000 (0:00:01.024) 0:01:19.014 ***** 2020-11-19T02:47:53.035 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-19T02:47:53.037 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:53.153 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYhJWrxQ5m7CqTHPhgzieL3qKuq90TOT4X'], 'uuids': ['830a44af-55bd-4af3-93f6-8e82e8e3b8ea']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.147516", "end": "2020-11-19 02:47:53.229800", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYhJWrxQ5m7CqTHPhgzieL3qKuq90TOT4X"], "labels": [], "masters": [], "uuids": ["830a44af-55bd-4af3-93f6-8e82e8e3b8ea"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:53.082284", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144128 s, 800 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144128 s, 800 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:53.155 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:53.166 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:53.320 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MKapE0CsIfWQuqdmtZsfd7daOflNOJUAo'], 'uuids': ['3312f5d4-6c50-4f49-baac-14cbe37be913']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.153310", "end": "2020-11-19 02:47:53.054185", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MKapE0CsIfWQuqdmtZsfd7daOflNOJUAo"], "labels": [], "masters": [], "uuids": ["3312f5d4-6c50-4f49-baac-14cbe37be913"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:52.900875", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150373 s, 767 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150373 s, 767 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:53.329 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:53.341 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:53.443 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYupgYo0wWdYRsa2v0ivQjfpc6nsRa9M6D'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156484", "end": "2020-11-19 02:47:53.529293", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYupgYo0wWdYRsa2v0ivQjfpc6nsRa9M6D"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:53.372809", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152493 s, 756 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152493 s, 756 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:53.457 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:53.615 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3McSr5kqzzAbN62mS4i0tNjjsdlEUsRNpZ'], 'uuids': ['7fc7a097-9dd3-466d-be50-06e50161c171']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.155285", "end": "2020-11-19 02:47:53.359895", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3McSr5kqzzAbN62mS4i0tNjjsdlEUsRNpZ"], "labels": [], "masters": [], "uuids": ["7fc7a097-9dd3-466d-be50-06e50161c171"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:53.204610", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152316 s, 757 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152316 s, 757 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:53.627 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFZB', 'wwn-0x5000c5009292466b'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFZB-part1', 'wwn-0x5000c5009292466b-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009292466b', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFZB', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFZB", "wwn-0x5000c5009292466b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFZB-part1", "wwn-0x5000c5009292466b-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFZB", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292466b"}}, "skip_reason": 2020-11-19T02:47:53.629 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-19T02:47:53.757 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYNhCEwiEGRj6rMOWhng19Wtw97vmriIY2'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.163273", "end": "2020-11-19 02:47:53.836687", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-uoJFeeOdxAKZnvd0qZkEgpjut2rXKegYNhCEwiEGRj6rMOWhng19Wtw97vmriIY2"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:53.673414", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158685 s, 727 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158685 s, 727 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:53.805 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:53 +0000 (0:00:00.831) 0:01:19.845 ***** 2020-11-19T02:47:53.816 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-0GNJMp-Ir33-1IFl-74cc-ys8I-dFk4-KflDjO', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300JY400BGN', 'nvme-nvme.8086-435646543632333330304a5934303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.054384", "end": "2020-11-19 02:47:53.548287", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-0GNJMp-Ir33-1IFl-74cc-ys8I-dFk4-KflDjO", "nvme-INTEL_SSDPEDMD400G4_CVFT623300JY400BGN", "nvme-nvme.8086-435646543632333330304a5934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:53.493903", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0496507 s, 2.3 GB/s", "std 2020-11-19T02:47:53.818 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0496507 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:53.831 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:54.075 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi086.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.126990", "end": "2020-11-19 02:47:54.152128", "rc": 0, "start": "2020-11-19 02:47:54.025138", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:54.116 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.309) 0:01:20.154 ***** 2020-11-19T02:47:54.124 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-19T02:47:54.131 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MQcKSk194GlrtHrJWkiZyI6QWX2aM6CKY'], 'uuids': ['600855cd-7ba5-48c4-a8bf-40fb7bea3afa']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.148827", "end": "2020-11-19 02:47:53.835657", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MQcKSk194GlrtHrJWkiZyI6QWX2aM6CKY"], "labels": [], "masters": [], "uuids": ["600855cd-7ba5-48c4-a8bf-40fb7bea3afa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:53.686830", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144446 s, 799 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144446 s, 799 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': ' 2020-11-19T02:47:54.134 INFO:teuthology.task.ansible.out:0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:54.332 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-19T02:47:54.334 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.051198", "end": "2020-11-19 02:47:54.425291", "rc": 0, "start": "2020-11-19 02:47:54.374093", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:54.367 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.253) 0:01:20.408 ***** 2020-11-19T02:47:54.396 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-19T02:47:54.433 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.065) 0:01:20.473 ***** 2020-11-19T02:47:54.440 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-19T02:47:54.442 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MtMx1s8E8BDDxpYOzypawpwZ70WCfLu81'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156464", "end": "2020-11-19 02:47:54.150624", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MtMx1s8E8BDDxpYOzypawpwZ70WCfLu81"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:53.994160", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151738 s, 760 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151738 s, 760 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:54.443 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": 2020-11-19T02:47:54.445 INFO:teuthology.task.ansible.out:{"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:54.465 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:54.506 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.072) 0:01:20.546 ***** 2020-11-19T02:47:54.543 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-19T02:47:54.546 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:54.582 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.075) 0:01:20.621 ***** 2020-11-19T02:47:54.607 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-19T02:47:54.608 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:54.645 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.064) 0:01:20.685 ***** 2020-11-19T02:47:54.675 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:54.716 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.070) 0:01:20.756 ***** 2020-11-19T02:47:54.723 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-19T02:47:54.724 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MHti4zBRFjircGffvVPJi7RSUGt3ftmGB'], 'uuids': ['2877b0c1-c746-4d42-af1a-c50e1e92f037']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.153400", "end": "2020-11-19 02:47:54.433425", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-quJkpOdHHYIGT25eDe4ZqDtOsESd0m3MHti4zBRFjircGffvVPJi7RSUGt3ftmGB"], "labels": [], "masters": [], "uuids": ["2877b0c1-c746-4d42-af1a-c50e1e92f037"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 02:47:54.280025", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148679 s, 776 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148679 s, 776 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:54.796 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.079) 0:01:20.836 ***** 2020-11-19T02:47:54.837 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T02:47:54.874 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.077) 0:01:20.914 ***** 2020-11-19T02:47:54.913 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi086.front.sepia.ceph.com] => (item={'block_used': 13753454, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341654, 'size_available': 927095414784, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341654, "block_size": 4096, "block_total": 240095108, "block_used": 13753454, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095414784, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:54.970 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:54 +0000 (0:00:00.094) 0:01:21.008 ***** 2020-11-19T02:47:54.998 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:55.044 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.076) 0:01:21.084 ***** 2020-11-19T02:47:55.077 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:47:55.079 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi160.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.132354", "end": "2020-11-19 02:47:54.808583", "rc": 0, "start": "2020-11-19 02:47:54.676229", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:55.123 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.078) 0:01:21.162 ***** 2020-11-19T02:47:55.165 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.043) 0:01:21.206 ***** 2020-11-19T02:47:55.172 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:47:55.174 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:55.220 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.053) 0:01:21.259 ***** 2020-11-19T02:47:55.247 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-19T02:47:55.249 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:55.283 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.063) 0:01:21.323 ***** 2020-11-19T02:47:55.326 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:55.368 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.085) 0:01:21.409 ***** 2020-11-19T02:47:55.375 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-19T02:47:55.377 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.052593", "end": "2020-11-19 02:47:55.101947", "rc": 0, "start": "2020-11-19 02:47:55.049354", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:47:55.425 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.056) 0:01:21.465 ***** 2020-11-19T02:47:55.432 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:55.473 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.047) 0:01:21.513 ***** 2020-11-19T02:47:55.480 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-19T02:47:55.522 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.048) 0:01:21.562 ***** 2020-11-19T02:47:55.555 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-19T02:47:55.558 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:55.601 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.079) 0:01:21.642 ***** 2020-11-19T02:47:55.631 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-19T02:47:55.633 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:55.667 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.065) 0:01:21.707 ***** 2020-11-19T02:47:55.694 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-19T02:47:55.697 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:55.732 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.065) 0:01:21.773 ***** 2020-11-19T02:47:55.767 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:55.812 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.078) 0:01:21.851 ***** 2020-11-19T02:47:55.949 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi160.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T02:47:55.987 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:55 +0000 (0:00:00.175) 0:01:22.027 ***** 2020-11-19T02:47:56.024 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-19T02:47:56.026 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'block_used': 13753457, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341651, 'size_available': 927095402496, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341651, "block_size": 4096, "block_total": 240095108, "block_used": 13753457, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095402496, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:56.061 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:56 +0000 (0:00:00.073) 0:01:22.101 ***** 2020-11-19T02:47:56.077 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-19T02:47:56.079 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-19T02:47:56.114 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:56 +0000 (0:00:00.053) 0:01:22.154 ***** 2020-11-19T02:47:56.121 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:56.158 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:56 +0000 (0:00:00.043) 0:01:22.197 ***** 2020-11-19T02:47:56.195 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:47:56.197 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:56.238 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:56 +0000 (0:00:00.081) 0:01:22.279 ***** 2020-11-19T02:47:56.266 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:47:56.268 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:56.302 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:56 +0000 (0:00:00.063) 0:01:22.343 ***** 2020-11-19T02:47:56.343 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:56.388 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:56 +0000 (0:00:00.086) 0:01:22.429 ***** 2020-11-19T02:47:56.417 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-19T02:47:56.419 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:56.462 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:56 +0000 (0:00:00.073) 0:01:22.502 ***** 2020-11-19T02:47:56.490 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-19T02:47:56.492 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:56.530 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:56 +0000 (0:00:00.068) 0:01:22.571 ***** 2020-11-19T02:47:56.666 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T02:47:56.886 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Create volume group(s)] *************************************** changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-19T02:47:56.923 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:56 +0000 (0:00:00.392) 0:01:22.963 ***** 2020-11-19T02:47:57.098 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T02:47:57.225 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T02:47:57.313 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T02:47:57.485 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T02:47:57.530 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T02:47:57.574 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:57 +0000 (0:00:00.651) 0:01:23.615 ***** 2020-11-19T02:47:57.723 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-19T02:47:57.724 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-19T02:47:57.762 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:57 +0000 (0:00:00.188) 0:01:23.803 ***** 2020-11-19T02:47:57.769 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-19T02:47:57.771 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T02:47:57.929 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-19T02:47:57.931 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T02:47:57.983 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-19T02:47:57.985 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T02:47:58.041 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T02:47:58.178 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T02:47:58.231 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-19T02:47:58.232 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T02:47:58.277 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:58 +0000 (0:00:00.512) 0:01:24.316 ***** 2020-11-19T02:47:58.304 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-19T02:47:58.306 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T02:47:58.317 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:58.359 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:58 +0000 (0:00:00.083) 0:01:24.399 ***** 2020-11-19T02:47:58.411 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-19T02:47:58.414 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-19T02:47:58.452 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:58 +0000 (0:00:00.092) 0:01:24.492 ***** 2020-11-19T02:47:58.503 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-11-19T02:47:58.505 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-19T02:47:58.542 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:58 +0000 (0:00:00.089) 0:01:24.582 ***** 2020-11-19T02:47:58.582 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-19T02:47:58.623 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:58 +0000 (0:00:00.078) 0:01:24.661 ***** 2020-11-19T02:47:58.630 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T02:47:58.731 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T02:47:58.851 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T02:47:58.960 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T02:47:58.971 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-19T02:47:59.015 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:59 +0000 (0:00:00.394) 0:01:25.056 ***** 2020-11-19T02:47:59.152 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-19T02:47:59.209 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:59 +0000 (0:00:00.184) 0:01:25.240 ***** 2020-11-19T02:47:59.241 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-19T02:47:59.286 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:59 +0000 (0:00:00.086) 0:01:25.327 ***** 2020-11-19T02:47:59.524 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-19T02:47:59.528 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true} 2020-11-19T02:47:59.561 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:47:59 +0000 (0:00:00.275) 0:01:25.602 ***** 2020-11-19T02:48:00.024 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-19T02:48:00.063 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:00 +0000 (0:00:00.501) 0:01:26.103 ***** 2020-11-19T02:48:00.105 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi086.front.sepia.ceph.com] => {"changed": true} 2020-11-19T02:48:00.141 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:00 +0000 (0:00:00.077) 0:01:26.181 ***** 2020-11-19T02:48:00.212 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-19T02:48:00.214 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-19T02:48:00.252 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:00 +0000 (0:00:00.110) 0:01:26.292 ***** 2020-11-19T02:48:00.322 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-19T02:48:00.324 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-19T02:48:00.359 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:00 +0000 (0:00:00.106) 0:01:26.399 ***** 2020-11-19T02:48:00.387 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-19T02:48:00.389 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-19T02:48:00.424 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:00 +0000 (0:00:00.065) 0:01:26.465 ***** 2020-11-19T02:48:00.517 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-19T02:48:00.553 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:00 +0000 (0:00:00.129) 0:01:26.594 ***** 2020-11-19T02:48:00.718 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-19T02:48:00.723 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-19T02:48:00.764 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:00 +0000 (0:00:00.210) 0:01:26.804 ***** 2020-11-19T02:48:00.773 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-19T02:48:00.775 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-19T02:48:00.823 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:00 +0000 (0:00:00.058) 0:01:26.863 ***** 2020-11-19T02:48:01.143 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-19T02:48:01.185 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:01 +0000 (0:00:00.362) 0:01:27.225 ***** 2020-11-19T02:48:01.194 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-19T02:48:01.196 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-19T02:48:01.241 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:01 +0000 (0:00:00.056) 0:01:27.282 ***** 2020-11-19T02:48:01.547 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-19T02:48:01.596 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:01 +0000 (0:00:00.354) 0:01:27.637 ***** 2020-11-19T02:48:02.159 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-19T02:48:02.161 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:48:02.198 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:02 +0000 (0:00:00.602) 0:01:28.239 ***** 2020-11-19T02:48:02.373 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-19T02:48:02.375 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046643", "end": "2020-11-19 02:48:02.127285", "rc": 0, "start": "2020-11-19 02:48:02.080642", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-19T02:48:02.409 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:02 +0000 (0:00:00.211) 0:01:28.450 ***** 2020-11-19T02:48:02.437 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-19T02:48:02.439 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:02.483 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:02 +0000 (0:00:00.072) 0:01:28.523 ***** 2020-11-19T02:48:02.491 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-19T02:48:02.493 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:48:02.537 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:02 +0000 (0:00:00.054) 0:01:28.577 ***** 2020-11-19T02:48:02.546 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-19T02:48:02.549 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:02.592 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:02 +0000 (0:00:00.054) 0:01:28.632 ***** 2020-11-19T02:48:02.793 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-19T02:48:02.795 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.053929", "end": "2020-11-19 02:48:02.877552", "rc": 0, "start": "2020-11-19 02:48:02.823623", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-19T02:48:02.833 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:02 +0000 (0:00:00.240) 0:01:28.873 ***** 2020-11-19T02:48:02.871 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-19T02:48:02.873 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:02.911 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:02 +0000 (0:00:00.077) 0:01:28.951 ***** 2020-11-19T02:48:02.938 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-19T02:48:02.940 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:02.976 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:02 +0000 (0:00:00.065) 0:01:29.016 ***** 2020-11-19T02:48:03.424 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-19T02:48:03.426 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:48:03.466 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:03 +0000 (0:00:00.489) 0:01:29.506 ***** 2020-11-19T02:48:03.493 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:03.529 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:03 +0000 (0:00:00.063) 0:01:29.569 ***** 2020-11-19T02:48:03.848 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-19T02:48:03.851 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:48:03.886 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:03 +0000 (0:00:00.357) 0:01:29.927 ***** 2020-11-19T02:48:03.922 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:03 +0000 (0:00:00.034) 0:01:29.962 ***** 2020-11-19T02:48:03.928 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-19T02:48:03.929 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:03.966 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:03 +0000 (0:00:00.044) 0:01:30.007 ***** 2020-11-19T02:48:03.973 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-19T02:48:03.975 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:04.030 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.063) 0:01:30.070 ***** 2020-11-19T02:48:04.237 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-19T02:48:04.239 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 02:46:29 UTC", "ActiveEnterTimestampMonotonic": "26844357", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target systemd-journald.socket network.target systemd-tmpfiles-setup.service -.mount system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 02:46:23 UTC", "AssertTimestampMonotonic": "20324132", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 02:46:23 UTC", "ConditionTimestampMonotonic": "20324132", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-19T02:48:04.240 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1856", "ExecMainStartTimestamp": "Thu 2020-11-19 02:46:29 UTC", "ExecMainStartTimestampMonotonic": "26844343", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 02:46:23 UTC", "InactiveExitTimestampMonotonic": "20325273", "InvocationID": "32337021d595412d990218b0268f854b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-19T02:48:04.242 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1856", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 02:46:29 UTC", "S 2020-11-19T02:48:04.244 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "26844357", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 02:46:29 UTC", "WatchdogTimestampMonotonic": "26844355", "WatchdogUSec": "0"}} 2020-11-19T02:48:04.280 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.249) 0:01:30.320 ***** 2020-11-19T02:48:04.299 INFO:teuthology.task.ansible.out: 2020-11-19T02:48:04.302 INFO:teuthology.task.ansible.out:TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} TASK [testnode : set_fact] ***************************************************** ok: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-19T02:48:04.341 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.059) 0:01:30.380 ***** 2020-11-19T02:48:04.377 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.037) 0:01:30.417 ***** 2020-11-19T02:48:04.382 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-19T02:48:04.384 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:04.419 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.043) 0:01:30.460 ***** 2020-11-19T02:48:04.426 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:48:04.427 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:04.463 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.043) 0:01:30.503 ***** 2020-11-19T02:48:04.490 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-19T02:48:04.491 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:04.531 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.068) 0:01:30.572 ***** 2020-11-19T02:48:04.625 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-19T02:48:04.627 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 02:46:17 UTC", "ActiveEnterTimestampMonotonic": "26740506", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket systemd-tmpfiles-setup.service system.slice basic.target sysinit.target network.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 02:46:10 UTC", "AssertTimestampMonotonic": "20195368", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 02:46:10 UTC", "ConditionTimestampMonotonic": "20195368", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-19T02:48:04.629 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1632", "ExecMainStartTimestamp": "Thu 2020-11-19 02:46:17 UTC", "ExecMainStartTimestampMonotonic": "26740492", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 02:46:10 UTC", "InactiveExitTimestampMonotonic": "20196561", "InvocationID": "86de3986e13b4cdd8daac6aec750d357", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-19T02:48:04.632 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1632", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 02:46:17 UTC", "S 2020-11-19T02:48:04.634 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "26740506", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 02:46:17 UTC", "WatchdogTimestampMonotonic": "26740504", "WatchdogUSec": "0"}} 2020-11-19T02:48:04.665 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.133) 0:01:30.705 ***** 2020-11-19T02:48:04.692 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:48:04.694 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-19T02:48:04.728 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.062) 0:01:30.768 ***** 2020-11-19T02:48:04.757 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:48:04.759 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:04.796 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.068) 0:01:30.836 ***** 2020-11-19T02:48:04.824 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-19T02:48:04.826 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:04.861 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:04 +0000 (0:00:00.065) 0:01:30.902 ***** 2020-11-19T02:48:05.362 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi160.front.sepia.ceph.com] => {"cache_update_time": 1605753819, "cache_updated": false, "changed": false} 2020-11-19T02:48:05.404 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:05 +0000 (0:00:00.541) 0:01:31.444 ***** 2020-11-19T02:48:05.431 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:05.467 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:05 +0000 (0:00:00.063) 0:01:31.508 ***** 2020-11-19T02:48:05.600 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-19T02:48:05.602 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-19T02:48:05.636 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:05 +0000 (0:00:00.169) 0:01:31.677 ***** 2020-11-19T02:48:05.921 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-19T02:48:05.923 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"cache_update_time": 1605753778, "cache_updated": false, "changed": false} 2020-11-19T02:48:05.982 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:05 +0000 (0:00:00.342) 0:01:32.019 ***** 2020-11-19T02:48:06.012 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-19T02:48:06.054 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:06 +0000 (0:00:00.074) 0:01:32.094 ***** 2020-11-19T02:48:06.060 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-19T02:48:06.062 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:06.101 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:06 +0000 (0:00:00.046) 0:01:32.141 ***** 2020-11-19T02:48:06.258 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-19T02:48:06.260 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-19T02:48:06.295 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:06 +0000 (0:00:00.195) 0:01:32.336 ***** 2020-11-19T02:48:06.613 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-19T02:48:06.648 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:06 +0000 (0:00:00.352) 0:01:32.688 ***** 2020-11-19T02:48:07.077 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-19T02:48:07.079 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-19T02:48:07.116 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:07 +0000 (0:00:00.468) 0:01:33.156 ***** 2020-11-19T02:48:07.413 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-19T02:48:07.468 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:07 +0000 (0:00:00.351) 0:01:33.508 ***** 2020-11-19T02:48:07.607 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-19T02:48:07.609 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false} 2020-11-19T02:48:07.655 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:07 +0000 (0:00:00.186) 0:01:33.696 ***** 2020-11-19T02:48:07.694 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-19T02:48:07.696 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-19T02:48:07.737 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:07 +0000 (0:00:00.082) 0:01:33.778 ***** 2020-11-19T02:48:07.883 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-19T02:48:07.885 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1605753756.8988104, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1605753756.8948104, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 2817, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1605753756.8948104, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2282523401", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-19T02:48:07.920 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:07 +0000 (0:00:00.182) 0:01:33.960 ***** 2020-11-19T02:48:08.139 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-19T02:48:08.141 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false} 2020-11-19T02:48:08.181 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.261) 0:01:34.222 ***** 2020-11-19T02:48:08.209 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-19T02:48:08.211 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-19T02:48:08.246 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.063) 0:01:34.286 ***** 2020-11-19T02:48:08.326 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** ok: [smithi160.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-19T02:48:08.366 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.120) 0:01:34.406 ***** 2020-11-19T02:48:08.397 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-19T02:48:08.398 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1605753743.0837262, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1605753743.0797262, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 2817, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1605753743.0797262, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2282523401", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-19T02:48:08.434 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.068) 0:01:34.475 ***** 2020-11-19T02:48:08.441 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-19T02:48:08.444 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-19T02:48:08.480 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.045) 0:01:34.521 ***** 2020-11-19T02:48:08.507 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-19T02:48:08.509 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:08.546 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.066) 0:01:34.587 ***** 2020-11-19T02:48:08.574 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-19T02:48:08.576 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-19T02:48:08.611 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.065) 0:01:34.652 ***** 2020-11-19T02:48:08.639 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-19T02:48:08.641 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-19T02:48:08.675 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.063) 0:01:34.715 ***** 2020-11-19T02:48:08.683 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi160.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-19T02:48:08.721 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.045) 0:01:34.760 ***** 2020-11-19T02:48:08.726 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-19T02:48:08.730 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:08.764 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.043) 0:01:34.804 ***** 2020-11-19T02:48:08.870 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-19T02:48:08.871 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-19T02:48:08.918 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.153) 0:01:34.958 ***** 2020-11-19T02:48:08.924 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-19T02:48:08.926 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-19T02:48:08.962 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:08 +0000 (0:00:00.044) 0:01:35.003 ***** 2020-11-19T02:48:08.968 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-19T02:48:08.971 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.010 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.047) 0:01:35.050 ***** 2020-11-19T02:48:09.048 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-19T02:48:09.050 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.059 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.061 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.071 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.072 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.115 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.105) 0:01:35.156 ***** 2020-11-19T02:48:09.122 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-19T02:48:09.123 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-19T02:48:09.160 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.044) 0:01:35.201 ***** 2020-11-19T02:48:09.169 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi160.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.171 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.181 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.183 INFO:teuthology.task.ansible.out: skipping: [smithi160.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi160.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.185 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.234 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.073) 0:01:35.275 ***** 2020-11-19T02:48:09.293 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.058) 0:01:35.334 ***** 2020-11-19T02:48:09.302 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-19T02:48:09.305 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.308 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.319 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.321 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.357 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.063) 0:01:35.398 ***** 2020-11-19T02:48:09.364 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:48:09.366 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.414 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.056) 0:01:35.454 ***** 2020-11-19T02:48:09.420 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-19T02:48:09.422 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.423 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditiona 2020-11-19T02:48:09.425 INFO:teuthology.task.ansible.out:l result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.460 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.046) 0:01:35.501 ***** 2020-11-19T02:48:09.467 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-19T02:48:09.469 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.505 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.044) 0:01:35.546 ***** 2020-11-19T02:48:09.511 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T02:48:09.514 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.550 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.045) 0:01:35.591 ***** 2020-11-19T02:48:09.557 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-19T02:48:09.560 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.597 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.045) 0:01:35.637 ***** 2020-11-19T02:48:09.603 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-19T02:48:09.605 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.640 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.043) 0:01:35.681 ***** 2020-11-19T02:48:09.648 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-19T02:48:09.650 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.684 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.044) 0:01:35.725 ***** 2020-11-19T02:48:09.690 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-19T02:48:09.692 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.730 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.045) 0:01:35.770 ***** 2020-11-19T02:48:09.736 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-19T02:48:09.738 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.778 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.048) 0:01:35.819 ***** 2020-11-19T02:48:09.785 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-19T02:48:09.787 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.822 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.042) 0:01:35.862 ***** 2020-11-19T02:48:09.830 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-19T02:48:09.831 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.868 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.046) 0:01:35.908 ***** 2020-11-19T02:48:09.875 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-19T02:48:09.876 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.922 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.053) 0:01:35.962 ***** 2020-11-19T02:48:09.935 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-19T02:48:09.937 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:09.975 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:09 +0000 (0:00:00.053) 0:01:36.016 ***** 2020-11-19T02:48:09.982 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-19T02:48:09.984 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:10.025 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:10 +0000 (0:00:00.049) 0:01:36.065 ***** 2020-11-19T02:48:10.031 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-19T02:48:10.033 INFO:teuthology.task.ansible.out:skipping: [smithi160.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:10.071 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:10 +0000 (0:00:00.046) 0:01:36.111 ***** 2020-11-19T02:48:10.077 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-19T02:48:10.079 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:10.117 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:10 +0000 (0:00:00.046) 0:01:36.158 ***** 2020-11-19T02:48:10.145 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T02:48:10.180 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:10 +0000 (0:00:00.063) 0:01:36.221 ***** 2020-11-19T02:48:10.229 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-19T02:48:10.235 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011129", "end": "2020-11-19 02:48:09.982938", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-19 02:48:09.971809", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:48:10.337 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.007190", "end": "2020-11-19 02:48:10.402464", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-19 02:48:10.395274", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:48:10.374 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006018", "end": "2020-11-19 02:48:10.116382", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-19 02:48:10.110364", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:48:10.451 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005994", "end": "2020-11-19 02:48:10.524517", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-19 02:48:10.518523", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:48:10.484 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005601", "end": "2020-11-19 02:48:10.234582", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-19 02:48:10.228981", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-19T02:48:10.579 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005423", "end": "2020-11-19 02:48:10.657457", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-19 02:48:10.652034", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-19T02:48:10.622 INFO:teuthology.task.ansible.out:ok: [smithi160.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.021897", "end": "2020-11-19 02:48:10.366633", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-19 02:48:10.344736", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:48:10.672 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:10 +0000 (0:00:00.491) 0:01:36.712 ***** 2020-11-19T02:48:10.700 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-19T02:48:10.752 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:10 +0000 (0:00:00.080) 0:01:36.793 ***** 2020-11-19T02:48:10.760 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-19T02:48:10.762 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026459", "end": "2020-11-19 02:48:10.811099", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-19 02:48:10.784640", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T02:48:10.818 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:10 +0000 (0:00:00.065) 0:01:36.859 ***** 2020-11-19T02:48:10.847 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-19T02:48:10.886 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 02:48:10 +0000 (0:00:00.068) 0:01:36.927 ***** 2020-11-19T02:48:10.893 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-19T02:48:10.895 INFO:teuthology.task.ansible.out:changed: [smithi160.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-19T02:48:11.020 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-19T02:48:11.128 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-19T02:48:11.132 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-19T02:48:11.136 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi086.front.sepia.ceph.com : ok=108 changed=27 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 smithi160.front.sepia.ceph.com : ok=108 changed=27 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 Thursday 19 November 2020 02:48:11 +0000 (0:00:00.250) 0:01:37.177 ***** =============================================================================== 2020-11-19T02:48:11.138 INFO:teuthology.task.ansible.out:testnode : Ensure packages are not present. ----------------------------- 8.24s testnode : Install packages --------------------------------------------- 7.98s testnode : Zap all non-root disks --------------------------------------- 7.60s common : Update apt cache ----------------------------------------------- 3.40s testnode : Update apt cache. -------------------------------------------- 2.75s testnode : Install apt keys --------------------------------------------- 2.14s Gathering Facts --------------------------------------------------------- 1.80s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 0.97s testnode : Install python-apt ------------------------------------------- 0.93s common : Install Kerberos Packages (Debian) ----------------------------- 0.89s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 0.88s testnode : Set the system's hostname ------------------------------------ 0.84s common : Install nrpe package and dependencies (Ubuntu/Fedora) ---------- 0.83s testnode : Blow away lingering OSD data and FSIDs ----------------------- 0.83s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.83s common : Make sure smartmontools is installed --------------------------- 0.80s users : Create all admin users with sudo access. ------------------------ 0.78s Gathering Facts --------------------------------------------------------- 0.69s ansible-managed : Add authorized keys for the ansible user. ------------- 0.65s testnode : Create logical volume(s) ------------------------------------- 0.65s 2020-11-19T02:48:11.506 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2020-11-19T02:48:11.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T02:48:11.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:48:11.574 INFO:teuthology.orchestra.run.smithi086:> true 2020-11-19T02:48:11.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T02:48:11.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T02:48:11.845 INFO:teuthology.orchestra.run.smithi160:> true 2020-11-19T02:48:12.048 INFO:teuthology.run_tasks:Running task clock... 2020-11-19T02:48:12.072 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-19T02:48:12.073 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-19T02:48:12.074 INFO:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-19T02:48:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-19T02:48:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: Command line: ntpd -gq 2020-11-19T02:48:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: proto: precision = 0.053 usec (-24) 2020-11-19T02:48:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-19T02:48:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: restrict ::: KOD does nothing without LIMITED. 2020-11-19T02:48:12.143 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: Listen and drop on 0 v6wildcard [::]:123 2020-11-19T02:48:12.143 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-19T02:48:12.143 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: Listen normally on 2 lo 127.0.0.1:123 2020-11-19T02:48:12.143 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: Listen normally on 3 enp3s0f1 172.21.15.86:123 2020-11-19T02:48:12.143 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: Listen normally on 4 lo [::1]:123 2020-11-19T02:48:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-19T02:48:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-19T02:48:12.145 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d809%5]:123 2020-11-19T02:48:12.145 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:12 ntpd[5588]: Listening on routing socket on fd #22 for interface updates 2020-11-19T02:48:13.142 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:13 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:13.143 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:13 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:13.143 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:13 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:13.143 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:13 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:15.143 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:15 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:15.144 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:15 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:15.144 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:15 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:15.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:15 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:15.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:15 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:15.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:15 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:15.146 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:15 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:15.146 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:15 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:16.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:16 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:16.146 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:16 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:16.147 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:16 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:16.148 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:16 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:16.148 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:16 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:16.149 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:16 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:16.149 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:16 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:16.150 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:16 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:17 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:17 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:17 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:17 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:17 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:17 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:17 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:17 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:18.144 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:18 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:18.144 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:18 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:18.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:18 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:18.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:18 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:18.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:18 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:18.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:18 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:18.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:18 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:18.146 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:18 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:19.144 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:19 ntpd[5588]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:19.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:19 ntpd[5588]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:19.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:19 ntpd[5588]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:19.145 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:19 ntpd[5588]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:19.146 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:19 ntpd[5588]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-19T02:48:19.146 INFO:teuthology.orchestra.run.smithi086.stderr:19 Nov 02:48:19 ntpd[5588]: can't open /var/log/ntpstats/loopstats.20201119: Permission denied 2020-11-19T02:48:19.146 INFO:teuthology.orchestra.run.smithi086.stdout:19 Nov 02:48:19 ntpd[5588]: ntpd: time set -0.098943 s 2020-11-19T02:48:19.146 INFO:teuthology.orchestra.run.smithi086.stdout:ntpd: time set -0.098943s 2020-11-19T02:48:19.211 INFO:teuthology.orchestra.run.smithi086.stdout: remote refid st t when poll reach delay offset jitter 2020-11-19T02:48:19.212 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================== 2020-11-19T02:48:19.212 INFO:teuthology.orchestra.run.smithi086.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T02:48:19.212 INFO:teuthology.orchestra.run.smithi086.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T02:48:19.212 INFO:teuthology.orchestra.run.smithi086.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T02:48:19.213 INFO:teuthology.orchestra.run.smithi086.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T02:48:19.214 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-19T02:48:19.214 INFO:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-19T02:48:19.270 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-19T02:48:19.271 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: Command line: ntpd -gq 2020-11-19T02:48:19.271 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: proto: precision = 0.123 usec (-23) 2020-11-19T02:48:19.272 INFO:teuthology.orchestra.run.smithi160.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-19T02:48:19.273 INFO:teuthology.orchestra.run.smithi160.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-19T02:48:19.273 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-19T02:48:19.273 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: restrict ::: KOD does nothing without LIMITED. 2020-11-19T02:48:19.273 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: Listen and drop on 0 v6wildcard [::]:123 2020-11-19T02:48:19.274 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-19T02:48:19.274 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: Listen normally on 2 lo 127.0.0.1:123 2020-11-19T02:48:19.274 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: Listen normally on 3 enp3s0f1 172.21.15.160:123 2020-11-19T02:48:19.274 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: Listen normally on 4 lo [::1]:123 2020-11-19T02:48:19.275 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f5b%5]:123 2020-11-19T02:48:19.275 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:19 ntpd[5712]: Listening on routing socket on fd #22 for interface updates 2020-11-19T02:48:20.274 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:20.275 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:20.276 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:20.276 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:20.277 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:20.278 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:20.278 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:20.278 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:20.279 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:20.279 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:20.285 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:20.286 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:20 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:22.271 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:22.271 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:22.271 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:22.271 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:22.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:22.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:22.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:22.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:22.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:22.273 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:22.273 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:22.273 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:22.273 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:22.273 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:22.274 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:22.274 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:22 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:24.270 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:24.270 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:24.270 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:24.270 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:24.271 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:24.271 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:24.271 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:24.271 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:24.271 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:24.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:24.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:24.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:24.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:24.272 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:24.273 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:24.273 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:24 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:26.269 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:27.331 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:27.332 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T02:48:27.332 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T02:48:27.332 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T02:48:27.332 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T02:48:27.333 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-19T02:48:27.333 INFO:teuthology.orchestra.run.smithi160.stderr:19 Nov 02:48:26 ntpd[5712]: can't open /var/log/ntpstats/loopstats.20201119: Permission denied 2020-11-19T02:48:27.333 INFO:teuthology.orchestra.run.smithi160.stdout:19 Nov 02:48:26 ntpd[5712]: ntpd: time set +0.211519 s 2020-11-19T02:48:27.333 INFO:teuthology.orchestra.run.smithi160.stdout:ntpd: time set +0.211519s 2020-11-19T02:48:27.334 INFO:teuthology.orchestra.run.smithi160.stdout: remote refid st t when poll reach delay offset jitter 2020-11-19T02:48:27.335 INFO:teuthology.orchestra.run.smithi160.stdout:============================================================================== 2020-11-19T02:48:27.335 INFO:teuthology.orchestra.run.smithi160.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T02:48:27.335 INFO:teuthology.orchestra.run.smithi160.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T02:48:27.335 INFO:teuthology.orchestra.run.smithi160.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T02:48:27.336 INFO:teuthology.orchestra.run.smithi160.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T02:48:27.336 INFO:teuthology.run_tasks:Running task install... 2020-11-19T02:48:55.033 DEBUG:teuthology.task.install:project ceph 2020-11-19T02:48:55.034 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}} 2020-11-19T02:48:55.034 DEBUG:teuthology.task.install:config {'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'} 2020-11-19T02:48:55.035 INFO:teuthology.task.install:Using flavor: basic 2020-11-19T02:48:55.044 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-19T02:48:55.045 INFO:teuthology.task.install:extra packages: [] 2020-11-19T02:48:55.045 INFO:teuthology.orchestra.run.smithi086:> sudo lsb_release -is 2020-11-19T02:48:55.101 INFO:teuthology.orchestra.run.smithi086.stdout:Ubuntu 2020-11-19T02:48:55.107 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T02:48:55.109 INFO:teuthology.orchestra.run.smithi160:> sudo lsb_release -is 2020-11-19T02:48:55.113 INFO:teuthology.orchestra.run.smithi086:> sudo apt-key list | grep Ceph 2020-11-19T02:48:55.171 INFO:teuthology.orchestra.run.smithi160.stdout:Ubuntu 2020-11-19T02:48:55.180 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T02:48:55.180 INFO:teuthology.orchestra.run.smithi160:> sudo apt-key list | grep Ceph 2020-11-19T02:48:55.218 INFO:teuthology.orchestra.run.smithi086.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-19T02:48:55.275 INFO:teuthology.orchestra.run.smithi160.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-19T02:48:55.280 INFO:teuthology.orchestra.run.smithi086.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-19T02:48:55.280 INFO:teuthology.orchestra.run.smithi086.stdout:uid [ unknown] Ceph.com (release key) 2020-11-19T02:48:55.281 INFO:teuthology.orchestra.run.smithi086:> sudo lsb_release -is 2020-11-19T02:48:55.331 INFO:teuthology.orchestra.run.smithi086.stdout:Ubuntu 2020-11-19T02:48:55.335 INFO:teuthology.orchestra.run.smithi160.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-19T02:48:55.335 INFO:teuthology.orchestra.run.smithi160.stdout:uid [ unknown] Ceph.com (release key) 2020-11-19T02:48:55.336 INFO:teuthology.orchestra.run.smithi160:> sudo lsb_release -is 2020-11-19T02:48:55.337 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T02:48:55.338 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-19T02:48:55.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T02:48:55.395 INFO:teuthology.orchestra.run.smithi160.stdout:Ubuntu 2020-11-19T02:48:55.401 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T02:48:55.402 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-19T02:48:55.402 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T02:48:55.530 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default/ 2020-11-19T02:48:55.530 INFO:teuthology.task.install.deb:Package version is 16.0.0-7199-g409eac50-1bionic 2020-11-19T02:48:55.576 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default/ 2020-11-19T02:48:55.577 INFO:teuthology.task.install.deb:Package version is 16.0.0-7199-g409eac50-1bionic 2020-11-19T02:48:55.667 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:48:55.668 INFO:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-19T02:48:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-19T02:48:55.687 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:48:55.687 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:48:55.688 INFO:teuthology.orchestra.run.smithi086.stderr:155 bytes copied, 7.5514e-05 s, 2.1 MB/s 2020-11-19T02:48:55.689 INFO:teuthology.orchestra.run.smithi086:> sudo apt-get update 2020-11-19T02:48:55.704 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:48:55.705 INFO:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-19T02:48:55.711 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-19T02:48:55.721 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:48:55.721 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:48:55.721 INFO:teuthology.orchestra.run.smithi160.stderr:155 bytes copied, 7.21e-05 s, 2.1 MB/s 2020-11-19T02:48:55.722 INFO:teuthology.orchestra.run.smithi160:> sudo apt-get update 2020-11-19T02:48:55.875 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-19T02:48:55.910 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-19T02:48:55.969 INFO:teuthology.orchestra.run.smithi086.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic InRelease 2020-11-19T02:48:55.991 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T02:48:55.993 INFO:teuthology.orchestra.run.smithi086.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-19T02:48:55.998 INFO:teuthology.orchestra.run.smithi160.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic InRelease 2020-11-19T02:48:56.017 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-19T02:48:56.018 INFO:teuthology.orchestra.run.smithi086.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-19T02:48:56.022 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T02:48:56.042 INFO:teuthology.orchestra.run.smithi160.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-19T02:48:56.044 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-19T02:48:56.065 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,941 B] 2020-11-19T02:48:56.072 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-19T02:48:56.079 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,941 B] 2020-11-19T02:48:56.102 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-19T02:48:56.104 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-19T02:48:56.153 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-19T02:48:56.182 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-19T02:48:56.473 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 26.7 kB in 1s (46.3 kB/s) 2020-11-19T02:48:56.493 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 26.7 kB in 1s (47.6 kB/s) 2020-11-19T02:48:57.700 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2020-11-19T02:48:57.717 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2020-11-19T02:48:57.746 INFO:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7199-g409eac50-1bionic cephadm=16.0.0-7199-g409eac50-1bionic ceph-mds=16.0.0-7199-g409eac50-1bionic ceph-mgr=16.0.0-7199-g409eac50-1bionic ceph-common=16.0.0-7199-g409eac50-1bionic ceph-fuse=16.0.0-7199-g409eac50-1bionic ceph-test=16.0.0-7199-g409eac50-1bionic radosgw=16.0.0-7199-g409eac50-1bionic python3-rados=16.0.0-7199-g409eac50-1bionic python3-rgw=16.0.0-7199-g409eac50-1bionic python3-cephfs=16.0.0-7199-g409eac50-1bionic python3-rbd=16.0.0-7199-g409eac50-1bionic libcephfs2=16.0.0-7199-g409eac50-1bionic libcephfs-dev=16.0.0-7199-g409eac50-1bionic librados2=16.0.0-7199-g409eac50-1bionic librbd1=16.0.0-7199-g409eac50-1bionic rbd-fuse=16.0.0-7199-g409eac50-1bionic 2020-11-19T02:48:57.764 INFO:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7199-g409eac50-1bionic cephadm=16.0.0-7199-g409eac50-1bionic ceph-mds=16.0.0-7199-g409eac50-1bionic ceph-mgr=16.0.0-7199-g409eac50-1bionic ceph-common=16.0.0-7199-g409eac50-1bionic ceph-fuse=16.0.0-7199-g409eac50-1bionic ceph-test=16.0.0-7199-g409eac50-1bionic radosgw=16.0.0-7199-g409eac50-1bionic python3-rados=16.0.0-7199-g409eac50-1bionic python3-rgw=16.0.0-7199-g409eac50-1bionic python3-cephfs=16.0.0-7199-g409eac50-1bionic python3-rbd=16.0.0-7199-g409eac50-1bionic libcephfs2=16.0.0-7199-g409eac50-1bionic libcephfs-dev=16.0.0-7199-g409eac50-1bionic librados2=16.0.0-7199-g409eac50-1bionic librbd1=16.0.0-7199-g409eac50-1bionic rbd-fuse=16.0.0-7199-g409eac50-1bionic 2020-11-19T02:48:57.829 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2020-11-19T02:48:57.847 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2020-11-19T02:48:58.086 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2020-11-19T02:48:58.086 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2020-11-19T02:48:58.105 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2020-11-19T02:48:58.106 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2020-11-19T02:48:58.264 INFO:teuthology.orchestra.run.smithi086.stdout:The following additional packages will be installed: 2020-11-19T02:48:58.265 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-19T02:48:58.267 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-19T02:48:58.267 INFO:teuthology.orchestra.run.smithi086.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-19T02:48:58.268 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-19T02:48:58.268 INFO:teuthology.orchestra.run.smithi086.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-19T02:48:58.268 INFO:teuthology.orchestra.run.smithi086.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-19T02:48:58.268 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-19T02:48:58.268 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-19T02:48:58.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-19T02:48:58.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-19T02:48:58.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-19T02:48:58.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-19T02:48:58.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-19T02:48:58.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-19T02:48:58.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-19T02:48:58.271 INFO:teuthology.orchestra.run.smithi160.stdout:The following additional packages will be installed: 2020-11-19T02:48:58.271 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-19T02:48:58.272 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-19T02:48:58.272 INFO:teuthology.orchestra.run.smithi160.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-19T02:48:58.272 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-19T02:48:58.272 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-19T02:48:58.273 INFO:teuthology.orchestra.run.smithi086.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-19T02:48:58.273 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-19T02:48:58.273 INFO:teuthology.orchestra.run.smithi086.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-19T02:48:58.273 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-19T02:48:58.273 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-19T02:48:58.274 INFO:teuthology.orchestra.run.smithi086.stdout:Suggested packages: 2020-11-19T02:48:58.274 INFO:teuthology.orchestra.run.smithi086.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-19T02:48:58.274 INFO:teuthology.orchestra.run.smithi086.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-19T02:48:58.274 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-19T02:48:58.274 INFO:teuthology.orchestra.run.smithi086.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-19T02:48:58.275 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-19T02:48:58.275 INFO:teuthology.orchestra.run.smithi086.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-19T02:48:58.275 INFO:teuthology.orchestra.run.smithi086.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-19T02:48:58.275 INFO:teuthology.orchestra.run.smithi086.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-19T02:48:58.275 INFO:teuthology.orchestra.run.smithi086.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-19T02:48:58.276 INFO:teuthology.orchestra.run.smithi086.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-19T02:48:58.276 INFO:teuthology.orchestra.run.smithi086.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-19T02:48:58.276 INFO:teuthology.orchestra.run.smithi086.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-19T02:48:58.276 INFO:teuthology.orchestra.run.smithi086.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-19T02:48:58.276 INFO:teuthology.orchestra.run.smithi086.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-19T02:48:58.277 INFO:teuthology.orchestra.run.smithi086.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-19T02:48:58.278 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-19T02:48:58.278 INFO:teuthology.orchestra.run.smithi160.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-19T02:48:58.278 INFO:teuthology.orchestra.run.smithi160.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-19T02:48:58.278 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-19T02:48:58.279 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-19T02:48:58.279 INFO:teuthology.orchestra.run.smithi160.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-19T02:48:58.279 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-19T02:48:58.279 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-19T02:48:58.279 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-19T02:48:58.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-19T02:48:58.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-19T02:48:58.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-19T02:48:58.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-19T02:48:58.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-19T02:48:58.281 INFO:teuthology.orchestra.run.smithi160.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-19T02:48:58.281 INFO:teuthology.orchestra.run.smithi160.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-19T02:48:58.281 INFO:teuthology.orchestra.run.smithi160.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-19T02:48:58.281 INFO:teuthology.orchestra.run.smithi160.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-19T02:48:58.281 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-19T02:48:58.282 INFO:teuthology.orchestra.run.smithi160.stdout:Suggested packages: 2020-11-19T02:48:58.282 INFO:teuthology.orchestra.run.smithi160.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-19T02:48:58.282 INFO:teuthology.orchestra.run.smithi160.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-19T02:48:58.282 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-19T02:48:58.282 INFO:teuthology.orchestra.run.smithi160.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-19T02:48:58.283 INFO:teuthology.orchestra.run.smithi160.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-19T02:48:58.283 INFO:teuthology.orchestra.run.smithi160.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-19T02:48:58.283 INFO:teuthology.orchestra.run.smithi160.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-19T02:48:58.283 INFO:teuthology.orchestra.run.smithi160.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-19T02:48:58.283 INFO:teuthology.orchestra.run.smithi160.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-19T02:48:58.284 INFO:teuthology.orchestra.run.smithi160.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-19T02:48:58.284 INFO:teuthology.orchestra.run.smithi160.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-19T02:48:58.284 INFO:teuthology.orchestra.run.smithi160.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-19T02:48:58.284 INFO:teuthology.orchestra.run.smithi160.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-19T02:48:58.284 INFO:teuthology.orchestra.run.smithi160.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-19T02:48:58.285 INFO:teuthology.orchestra.run.smithi160.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-19T02:48:58.353 INFO:teuthology.orchestra.run.smithi086.stdout:The following NEW packages will be installed: 2020-11-19T02:48:58.353 INFO:teuthology.orchestra.run.smithi086.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-19T02:48:58.354 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-19T02:48:58.355 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-19T02:48:58.355 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-19T02:48:58.356 INFO:teuthology.orchestra.run.smithi086.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-19T02:48:58.356 INFO:teuthology.orchestra.run.smithi086.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-19T02:48:58.356 INFO:teuthology.orchestra.run.smithi086.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-19T02:48:58.356 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-19T02:48:58.356 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-19T02:48:58.357 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-19T02:48:58.357 INFO:teuthology.orchestra.run.smithi086.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-19T02:48:58.357 INFO:teuthology.orchestra.run.smithi086.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-19T02:48:58.357 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-19T02:48:58.357 INFO:teuthology.orchestra.run.smithi086.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-19T02:48:58.358 INFO:teuthology.orchestra.run.smithi086.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-19T02:48:58.358 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-19T02:48:58.358 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-19T02:48:58.358 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-19T02:48:58.358 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-19T02:48:58.359 INFO:teuthology.orchestra.run.smithi086.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-19T02:48:58.359 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-19T02:48:58.359 INFO:teuthology.orchestra.run.smithi086.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-19T02:48:58.359 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-19T02:48:58.359 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-19T02:48:58.360 INFO:teuthology.orchestra.run.smithi086.stdout:The following packages will be upgraded: 2020-11-19T02:48:58.363 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 librbd1 2020-11-19T02:48:58.374 INFO:teuthology.orchestra.run.smithi160.stdout:The following NEW packages will be installed: 2020-11-19T02:48:58.374 INFO:teuthology.orchestra.run.smithi160.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-19T02:48:58.374 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-19T02:48:58.375 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-19T02:48:58.376 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-19T02:48:58.376 INFO:teuthology.orchestra.run.smithi160.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-19T02:48:58.376 INFO:teuthology.orchestra.run.smithi160.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-19T02:48:58.376 INFO:teuthology.orchestra.run.smithi160.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-19T02:48:58.377 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-19T02:48:58.377 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-19T02:48:58.377 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-19T02:48:58.377 INFO:teuthology.orchestra.run.smithi160.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-19T02:48:58.378 INFO:teuthology.orchestra.run.smithi160.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-19T02:48:58.378 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-19T02:48:58.378 INFO:teuthology.orchestra.run.smithi160.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-19T02:48:58.378 INFO:teuthology.orchestra.run.smithi160.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-19T02:48:58.379 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-19T02:48:58.379 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-19T02:48:58.379 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-19T02:48:58.379 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-19T02:48:58.379 INFO:teuthology.orchestra.run.smithi160.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-19T02:48:58.380 INFO:teuthology.orchestra.run.smithi160.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-19T02:48:58.380 INFO:teuthology.orchestra.run.smithi160.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-19T02:48:58.380 INFO:teuthology.orchestra.run.smithi160.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-19T02:48:58.380 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-19T02:48:58.381 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages will be upgraded: 2020-11-19T02:48:58.381 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 librbd1 2020-11-19T02:48:58.517 INFO:teuthology.orchestra.run.smithi086.stdout:2 upgraded, 110 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T02:48:58.518 INFO:teuthology.orchestra.run.smithi086.stdout:Need to get 168 MB of archives. 2020-11-19T02:48:58.518 INFO:teuthology.orchestra.run.smithi086.stdout:After this operation, 782 MB of additional disk space will be used. 2020-11-19T02:48:58.518 INFO:teuthology.orchestra.run.smithi086.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7199-g409eac50-1bionic [1,924 kB] 2020-11-19T02:48:58.540 INFO:teuthology.orchestra.run.smithi160.stdout:2 upgraded, 110 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T02:48:58.541 INFO:teuthology.orchestra.run.smithi160.stdout:Need to get 168 MB of archives. 2020-11-19T02:48:58.541 INFO:teuthology.orchestra.run.smithi160.stdout:After this operation, 782 MB of additional disk space will be used. 2020-11-19T02:48:58.541 INFO:teuthology.orchestra.run.smithi160.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7199-g409eac50-1bionic [1,924 kB] 2020-11-19T02:48:58.542 INFO:teuthology.orchestra.run.smithi086.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-19T02:48:58.565 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-19T02:48:58.623 INFO:teuthology.orchestra.run.smithi086.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-19T02:48:58.646 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-19T02:48:58.712 INFO:teuthology.orchestra.run.smithi086.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7199-g409eac50-1bionic [3,160 kB] 2020-11-19T02:48:58.738 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7199-g409eac50-1bionic [3,160 kB] 2020-11-19T02:48:58.878 INFO:teuthology.orchestra.run.smithi086.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-19T02:48:58.887 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-19T02:48:58.897 INFO:teuthology.orchestra.run.smithi086.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-19T02:48:58.904 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7199-g409eac50-1bionic [478 kB] 2020-11-19T02:48:58.905 INFO:teuthology.orchestra.run.smithi160.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-19T02:48:58.919 INFO:teuthology.orchestra.run.smithi086.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7199-g409eac50-1bionic [332 kB] 2020-11-19T02:48:58.920 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-19T02:48:58.938 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7199-g409eac50-1bionic [478 kB] 2020-11-19T02:48:58.942 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-19T02:48:58.954 INFO:teuthology.orchestra.run.smithi160.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7199-g409eac50-1bionic [332 kB] 2020-11-19T02:48:58.963 INFO:teuthology.orchestra.run.smithi086.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7199-g409eac50-1bionic [20.6 kB] 2020-11-19T02:48:58.964 INFO:teuthology.orchestra.run.smithi086.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7199-g409eac50-1bionic [168 kB] 2020-11-19T02:48:58.969 INFO:teuthology.orchestra.run.smithi086.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7199-g409eac50-1bionic [27.7 kB] 2020-11-19T02:48:58.970 INFO:teuthology.orchestra.run.smithi086.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7199-g409eac50-1bionic [330 kB] 2020-11-19T02:48:58.981 INFO:teuthology.orchestra.run.smithi086.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7199-g409eac50-1bionic [3,178 kB] 2020-11-19T02:48:59.000 INFO:teuthology.orchestra.run.smithi086.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-19T02:48:59.000 INFO:teuthology.orchestra.run.smithi160.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7199-g409eac50-1bionic [20.6 kB] 2020-11-19T02:48:59.002 INFO:teuthology.orchestra.run.smithi160.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7199-g409eac50-1bionic [168 kB] 2020-11-19T02:48:59.003 INFO:teuthology.orchestra.run.smithi086.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-19T02:48:59.007 INFO:teuthology.orchestra.run.smithi160.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7199-g409eac50-1bionic [27.7 kB] 2020-11-19T02:48:59.007 INFO:teuthology.orchestra.run.smithi160.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7199-g409eac50-1bionic [330 kB] 2020-11-19T02:48:59.018 INFO:teuthology.orchestra.run.smithi160.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7199-g409eac50-1bionic [3,178 kB] 2020-11-19T02:48:59.035 INFO:teuthology.orchestra.run.smithi086.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-19T02:48:59.044 INFO:teuthology.orchestra.run.smithi086.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-19T02:48:59.050 INFO:teuthology.orchestra.run.smithi160.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-19T02:48:59.052 INFO:teuthology.orchestra.run.smithi086.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-19T02:48:59.053 INFO:teuthology.orchestra.run.smithi160.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-19T02:48:59.062 INFO:teuthology.orchestra.run.smithi086.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-19T02:48:59.072 INFO:teuthology.orchestra.run.smithi086.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-19T02:48:59.080 INFO:teuthology.orchestra.run.smithi160.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-19T02:48:59.088 INFO:teuthology.orchestra.run.smithi160.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-19T02:48:59.098 INFO:teuthology.orchestra.run.smithi160.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-19T02:48:59.102 INFO:teuthology.orchestra.run.smithi086.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-19T02:48:59.104 INFO:teuthology.orchestra.run.smithi086.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-19T02:48:59.111 INFO:teuthology.orchestra.run.smithi160.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-19T02:48:59.119 INFO:teuthology.orchestra.run.smithi160.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-19T02:48:59.119 INFO:teuthology.orchestra.run.smithi086.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-19T02:48:59.130 INFO:teuthology.orchestra.run.smithi086.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-19T02:48:59.131 INFO:teuthology.orchestra.run.smithi086.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-19T02:48:59.143 INFO:teuthology.orchestra.run.smithi160.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-19T02:48:59.147 INFO:teuthology.orchestra.run.smithi160.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-19T02:48:59.150 INFO:teuthology.orchestra.run.smithi160.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-19T02:48:59.164 INFO:teuthology.orchestra.run.smithi160.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-19T02:48:59.195 INFO:teuthology.orchestra.run.smithi086.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7199-g409eac50-1bionic [99.2 kB] 2020-11-19T02:48:59.198 INFO:teuthology.orchestra.run.smithi086.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-19T02:48:59.198 INFO:teuthology.orchestra.run.smithi086.stdout:Get:29 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7199-g409eac50-1bionic [414 kB] 2020-11-19T02:48:59.199 INFO:teuthology.orchestra.run.smithi086.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-19T02:48:59.201 INFO:teuthology.orchestra.run.smithi086.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-19T02:48:59.202 INFO:teuthology.orchestra.run.smithi086.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-19T02:48:59.202 INFO:teuthology.orchestra.run.smithi086.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-19T02:48:59.650 INFO:teuthology.orchestra.run.smithi160.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-19T02:48:59.650 INFO:teuthology.orchestra.run.smithi160.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-19T02:48:59.650 INFO:teuthology.orchestra.run.smithi160.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-19T02:48:59.651 INFO:teuthology.orchestra.run.smithi160.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-19T02:48:59.651 INFO:teuthology.orchestra.run.smithi160.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-19T02:48:59.651 INFO:teuthology.orchestra.run.smithi160.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-19T02:48:59.651 INFO:teuthology.orchestra.run.smithi160.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-19T02:48:59.651 INFO:teuthology.orchestra.run.smithi160.stdout:Get:33 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7199-g409eac50-1bionic [99.2 kB] 2020-11-19T02:48:59.652 INFO:teuthology.orchestra.run.smithi160.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7199-g409eac50-1bionic [414 kB] 2020-11-19T02:48:59.652 INFO:teuthology.orchestra.run.smithi160.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7199-g409eac50-1bionic [20.5 MB] 2020-11-19T02:48:59.652 INFO:teuthology.orchestra.run.smithi160.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-19T02:48:59.652 INFO:teuthology.orchestra.run.smithi160.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-19T02:48:59.652 INFO:teuthology.orchestra.run.smithi160.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-19T02:48:59.653 INFO:teuthology.orchestra.run.smithi160.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-19T02:48:59.653 INFO:teuthology.orchestra.run.smithi160.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-19T02:48:59.653 INFO:teuthology.orchestra.run.smithi160.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-19T02:48:59.653 INFO:teuthology.orchestra.run.smithi160.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-19T02:48:59.654 INFO:teuthology.orchestra.run.smithi160.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-19T02:48:59.654 INFO:teuthology.orchestra.run.smithi160.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-19T02:48:59.654 INFO:teuthology.orchestra.run.smithi160.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-19T02:48:59.654 INFO:teuthology.orchestra.run.smithi160.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-19T02:48:59.654 INFO:teuthology.orchestra.run.smithi160.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-19T02:48:59.654 INFO:teuthology.orchestra.run.smithi160.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-19T02:48:59.655 INFO:teuthology.orchestra.run.smithi160.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-19T02:48:59.655 INFO:teuthology.orchestra.run.smithi086.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-19T02:48:59.655 INFO:teuthology.orchestra.run.smithi086.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-19T02:48:59.655 INFO:teuthology.orchestra.run.smithi086.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7199-g409eac50-1bionic [20.5 MB] 2020-11-19T02:48:59.656 INFO:teuthology.orchestra.run.smithi086.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-19T02:48:59.656 INFO:teuthology.orchestra.run.smithi086.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-19T02:48:59.656 INFO:teuthology.orchestra.run.smithi086.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-19T02:48:59.656 INFO:teuthology.orchestra.run.smithi086.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-19T02:48:59.656 INFO:teuthology.orchestra.run.smithi086.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-19T02:48:59.657 INFO:teuthology.orchestra.run.smithi086.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-19T02:48:59.657 INFO:teuthology.orchestra.run.smithi086.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-19T02:48:59.657 INFO:teuthology.orchestra.run.smithi086.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-19T02:48:59.657 INFO:teuthology.orchestra.run.smithi086.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-19T02:48:59.657 INFO:teuthology.orchestra.run.smithi086.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-19T02:48:59.658 INFO:teuthology.orchestra.run.smithi086.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-19T02:48:59.658 INFO:teuthology.orchestra.run.smithi086.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-19T02:48:59.658 INFO:teuthology.orchestra.run.smithi086.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-19T02:48:59.797 INFO:teuthology.orchestra.run.smithi086.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-19T02:48:59.833 INFO:teuthology.orchestra.run.smithi160.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-19T02:48:59.841 INFO:teuthology.orchestra.run.smithi086.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-19T02:48:59.842 INFO:teuthology.orchestra.run.smithi086.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-19T02:48:59.876 INFO:teuthology.orchestra.run.smithi160.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-19T02:48:59.877 INFO:teuthology.orchestra.run.smithi086.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi160.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi086.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi086.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi086.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi086.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi086.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi086.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-19T02:48:59.914 INFO:teuthology.orchestra.run.smithi160.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-19T02:48:59.915 INFO:teuthology.orchestra.run.smithi160.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-19T02:48:59.915 INFO:teuthology.orchestra.run.smithi160.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-19T02:48:59.916 INFO:teuthology.orchestra.run.smithi160.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-19T02:48:59.917 INFO:teuthology.orchestra.run.smithi160.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-19T02:48:59.918 INFO:teuthology.orchestra.run.smithi160.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-19T02:48:59.919 INFO:teuthology.orchestra.run.smithi160.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-19T02:48:59.957 INFO:teuthology.orchestra.run.smithi086.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-19T02:48:59.958 INFO:teuthology.orchestra.run.smithi086.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-19T02:48:59.993 INFO:teuthology.orchestra.run.smithi160.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-19T02:48:59.994 INFO:teuthology.orchestra.run.smithi160.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-19T02:49:00.037 INFO:teuthology.orchestra.run.smithi086.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-19T02:49:00.038 INFO:teuthology.orchestra.run.smithi086.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi086.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi086.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi086.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-19T02:49:00.049 INFO:teuthology.orchestra.run.smithi086.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-19T02:49:00.049 INFO:teuthology.orchestra.run.smithi086.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-19T02:49:00.050 INFO:teuthology.orchestra.run.smithi086.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-19T02:49:00.073 INFO:teuthology.orchestra.run.smithi160.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-19T02:49:00.074 INFO:teuthology.orchestra.run.smithi160.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-19T02:49:00.079 INFO:teuthology.orchestra.run.smithi160.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-19T02:49:00.080 INFO:teuthology.orchestra.run.smithi160.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-19T02:49:00.081 INFO:teuthology.orchestra.run.smithi160.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-19T02:49:00.083 INFO:teuthology.orchestra.run.smithi160.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-19T02:49:00.084 INFO:teuthology.orchestra.run.smithi160.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-19T02:49:00.085 INFO:teuthology.orchestra.run.smithi160.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-19T02:49:00.116 INFO:teuthology.orchestra.run.smithi086.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-19T02:49:00.125 INFO:teuthology.orchestra.run.smithi086.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-19T02:49:00.153 INFO:teuthology.orchestra.run.smithi160.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-19T02:49:00.158 INFO:teuthology.orchestra.run.smithi160.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-19T02:49:00.196 INFO:teuthology.orchestra.run.smithi086.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-19T02:49:00.203 INFO:teuthology.orchestra.run.smithi086.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-19T02:49:00.204 INFO:teuthology.orchestra.run.smithi086.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-19T02:49:00.204 INFO:teuthology.orchestra.run.smithi086.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-19T02:49:00.205 INFO:teuthology.orchestra.run.smithi086.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-19T02:49:00.207 INFO:teuthology.orchestra.run.smithi086.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi086.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi086.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi160.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-19T02:49:00.235 INFO:teuthology.orchestra.run.smithi160.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-19T02:49:00.236 INFO:teuthology.orchestra.run.smithi160.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-19T02:49:00.236 INFO:teuthology.orchestra.run.smithi160.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-19T02:49:00.237 INFO:teuthology.orchestra.run.smithi160.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-19T02:49:00.238 INFO:teuthology.orchestra.run.smithi160.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-19T02:49:00.240 INFO:teuthology.orchestra.run.smithi160.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-19T02:49:00.241 INFO:teuthology.orchestra.run.smithi160.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-19T02:49:00.275 INFO:teuthology.orchestra.run.smithi086.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-19T02:49:00.276 INFO:teuthology.orchestra.run.smithi086.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-19T02:49:00.312 INFO:teuthology.orchestra.run.smithi160.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-19T02:49:00.313 INFO:teuthology.orchestra.run.smithi160.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-19T02:49:00.355 INFO:teuthology.orchestra.run.smithi086.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-19T02:49:00.378 INFO:teuthology.orchestra.run.smithi086.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-19T02:49:00.379 INFO:teuthology.orchestra.run.smithi086.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi086.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-19T02:49:00.392 INFO:teuthology.orchestra.run.smithi160.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-19T02:49:00.414 INFO:teuthology.orchestra.run.smithi160.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-19T02:49:00.415 INFO:teuthology.orchestra.run.smithi160.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-19T02:49:00.416 INFO:teuthology.orchestra.run.smithi160.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-19T02:49:00.501 INFO:teuthology.orchestra.run.smithi160.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-19T02:49:00.502 INFO:teuthology.orchestra.run.smithi160.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-19T02:49:00.503 INFO:teuthology.orchestra.run.smithi160.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-19T02:49:00.516 INFO:teuthology.orchestra.run.smithi086.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-19T02:49:00.517 INFO:teuthology.orchestra.run.smithi086.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-19T02:49:00.519 INFO:teuthology.orchestra.run.smithi086.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-19T02:49:00.525 INFO:teuthology.orchestra.run.smithi086.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-19T02:49:00.525 INFO:teuthology.orchestra.run.smithi086.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-19T02:49:00.527 INFO:teuthology.orchestra.run.smithi086.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-19T02:49:00.527 INFO:teuthology.orchestra.run.smithi086.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi086.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi086.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi086.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7199-g409eac50-1bionic [5,564 kB] 2020-11-19T02:49:00.560 INFO:teuthology.orchestra.run.smithi160.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-19T02:49:00.561 INFO:teuthology.orchestra.run.smithi160.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-19T02:49:00.562 INFO:teuthology.orchestra.run.smithi160.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-19T02:49:00.562 INFO:teuthology.orchestra.run.smithi160.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-19T02:49:00.564 INFO:teuthology.orchestra.run.smithi160.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-19T02:49:00.564 INFO:teuthology.orchestra.run.smithi160.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-19T02:49:00.640 INFO:teuthology.orchestra.run.smithi160.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7199-g409eac50-1bionic [5,564 kB] 2020-11-19T02:49:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7199-g409eac50-1bionic [165 kB] 2020-11-19T02:49:00.920 INFO:teuthology.orchestra.run.smithi086.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7199-g409eac50-1bionic [1,361 kB] 2020-11-19T02:49:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7199-g409eac50-1bionic [6,661 kB] 2020-11-19T02:49:01.037 INFO:teuthology.orchestra.run.smithi160.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7199-g409eac50-1bionic [165 kB] 2020-11-19T02:49:01.041 INFO:teuthology.orchestra.run.smithi160.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7199-g409eac50-1bionic [1,361 kB] 2020-11-19T02:49:01.122 INFO:teuthology.orchestra.run.smithi160.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7199-g409eac50-1bionic [6,661 kB] 2020-11-19T02:49:01.432 INFO:teuthology.orchestra.run.smithi086.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7199-g409eac50-1bionic [24.6 MB] 2020-11-19T02:49:01.565 INFO:teuthology.orchestra.run.smithi160.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7199-g409eac50-1bionic [24.6 MB] 2020-11-19T02:49:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7199-g409eac50-1bionic [3,816 B] 2020-11-19T02:49:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7199-g409eac50-1bionic [563 kB] 2020-11-19T02:49:03.052 INFO:teuthology.orchestra.run.smithi086.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7199-g409eac50-1bionic [1,941 kB] 2020-11-19T02:49:03.156 INFO:teuthology.orchestra.run.smithi086.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7199-g409eac50-1bionic [51.7 kB] 2020-11-19T02:49:03.156 INFO:teuthology.orchestra.run.smithi086.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7199-g409eac50-1bionic [56.2 kB] 2020-11-19T02:49:03.158 INFO:teuthology.orchestra.run.smithi086.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7199-g409eac50-1bionic [2,877 kB] 2020-11-19T02:49:03.212 INFO:teuthology.orchestra.run.smithi160.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7199-g409eac50-1bionic [3,816 B] 2020-11-19T02:49:03.213 INFO:teuthology.orchestra.run.smithi160.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7199-g409eac50-1bionic [563 kB] 2020-11-19T02:49:03.267 INFO:teuthology.orchestra.run.smithi160.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7199-g409eac50-1bionic [1,941 kB] 2020-11-19T02:49:03.334 INFO:teuthology.orchestra.run.smithi086.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7199-g409eac50-1bionic [7,885 kB] 2020-11-19T02:49:03.376 INFO:teuthology.orchestra.run.smithi160.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7199-g409eac50-1bionic [51.7 kB] 2020-11-19T02:49:03.412 INFO:teuthology.orchestra.run.smithi160.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7199-g409eac50-1bionic [56.2 kB] 2020-11-19T02:49:03.418 INFO:teuthology.orchestra.run.smithi160.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7199-g409eac50-1bionic [2,877 kB] 2020-11-19T02:49:03.596 INFO:teuthology.orchestra.run.smithi160.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7199-g409eac50-1bionic [7,885 kB] 2020-11-19T02:49:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7199-g409eac50-1bionic [3,932 B] 2020-11-19T02:49:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7199-g409eac50-1bionic [22.3 kB] 2020-11-19T02:49:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7199-g409eac50-1bionic [49.6 MB] 2020-11-19T02:49:04.041 INFO:teuthology.orchestra.run.smithi160.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7199-g409eac50-1bionic [3,932 B] 2020-11-19T02:49:04.041 INFO:teuthology.orchestra.run.smithi160.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7199-g409eac50-1bionic [22.3 kB] 2020-11-19T02:49:04.042 INFO:teuthology.orchestra.run.smithi160.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7199-g409eac50-1bionic [49.6 MB] 2020-11-19T02:49:06.954 INFO:teuthology.orchestra.run.smithi160.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7199-g409eac50-1bionic [19.5 kB] 2020-11-19T02:49:06.954 INFO:teuthology.orchestra.run.smithi160.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7199-g409eac50-1bionic [9,854 kB] 2020-11-19T02:49:07.550 INFO:teuthology.orchestra.run.smithi160.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7199-g409eac50-1bionic [76.9 kB] 2020-11-19T02:49:07.576 INFO:teuthology.orchestra.run.smithi086.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7199-g409eac50-1bionic [19.5 kB] 2020-11-19T02:49:07.576 INFO:teuthology.orchestra.run.smithi086.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7199-g409eac50-1bionic [9,854 kB] 2020-11-19T02:49:07.896 INFO:teuthology.orchestra.run.smithi086.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7199-g409eac50-1bionic [76.9 kB] 2020-11-19T02:49:08.256 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 168 MB in 9s (18.4 MB/s) 2020-11-19T02:49:08.402 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-19T02:49:08.453 INFO:teuthology.orchestra.run.smithi160.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137607 files and directories currently installed.) 2020-11-19T02:49:08.457 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-19T02:49:08.501 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-19T02:49:08.613 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 168 MB in 10s (17.7 MB/s) 2020-11-19T02:49:08.812 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-19T02:49:08.812 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-19T02:49:08.828 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-19T02:49:08.864 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137607 files and directories currently installed.) 2020-11-19T02:49:08.868 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-19T02:49:08.895 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-19T02:49:08.911 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-19T02:49:09.230 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-19T02:49:09.231 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../002-librbd1_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:09.244 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-19T02:49:09.331 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-19T02:49:09.357 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librbd1 (16.0.0-7199-g409eac50-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-19T02:49:09.657 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../002-librbd1_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:09.784 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librbd1 (16.0.0-7199-g409eac50-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-19T02:49:09.910 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../003-librados2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:10.053 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librados2 (16.0.0-7199-g409eac50-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-19T02:49:10.337 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../003-librados2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:10.463 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librados2 (16.0.0-7199-g409eac50-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-19T02:49:10.635 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libcephfs2. 2020-11-19T02:49:12.688 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libcephfs2. 2020-11-19T02:49:12.689 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:12.689 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libcephfs2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.690 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rados. 2020-11-19T02:49:12.690 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:12.690 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rados (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.691 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-19T02:49:12.691 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:12.691 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-ceph-argparse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.692 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cephfs. 2020-11-19T02:49:12.692 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:12.692 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cephfs (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.692 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-19T02:49:12.692 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:12.693 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.693 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:12.693 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libcephfs2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.694 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rados. 2020-11-19T02:49:12.694 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:12.694 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rados (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.694 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-19T02:49:12.695 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:12.695 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-ceph-argparse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.695 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cephfs. 2020-11-19T02:49:12.695 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:12.696 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cephfs (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.696 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-19T02:49:12.696 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:12.697 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.697 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-prettytable. 2020-11-19T02:49:12.697 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-19T02:49:12.698 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-19T02:49:12.780 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rbd. 2020-11-19T02:49:12.793 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:12.830 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rbd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:12.855 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-prettytable. 2020-11-19T02:49:12.868 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-19T02:49:12.896 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-19T02:49:13.124 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-19T02:49:13.139 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-19T02:49:13.141 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rbd. 2020-11-19T02:49:13.153 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:13.174 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-19T02:49:13.191 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rbd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:13.543 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-19T02:49:13.552 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-19T02:49:13.557 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-19T02:49:13.565 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-19T02:49:13.602 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-19T02:49:13.610 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-19T02:49:13.904 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-19T02:49:13.919 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-19T02:49:13.946 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librgw2. 2020-11-19T02:49:13.962 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../013-librgw2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:13.963 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-19T02:49:14.004 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librgw2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:14.282 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librgw2. 2020-11-19T02:49:14.296 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../013-librgw2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:14.362 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librgw2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:14.557 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rgw. 2020-11-19T02:49:14.571 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:14.608 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:14.876 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-19T02:49:14.891 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-19T02:49:14.910 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rgw. 2020-11-19T02:49:14.922 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:14.926 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-19T02:49:14.959 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:15.237 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-19T02:49:15.251 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-19T02:49:15.279 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-19T02:49:15.292 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-19T02:49:15.296 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-19T02:49:15.337 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-19T02:49:15.681 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package liboath0. 2020-11-19T02:49:15.696 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-19T02:49:15.740 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-19T02:49:15.740 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-19T02:49:15.755 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-19T02:49:15.790 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-19T02:49:16.017 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libradosstriper1. 2020-11-19T02:49:16.030 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:16.059 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libradosstriper1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:16.159 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package liboath0. 2020-11-19T02:49:16.173 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-19T02:49:16.226 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-19T02:49:16.520 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libradosstriper1. 2020-11-19T02:49:16.536 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:16.554 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-common. 2020-11-19T02:49:16.568 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:16.579 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libradosstriper1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:16.612 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:16.982 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-common. 2020-11-19T02:49:16.995 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:17.023 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:18.459 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-base. 2020-11-19T02:49:18.473 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:18.526 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-base (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:18.937 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-base. 2020-11-19T02:49:18.951 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:19.004 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-base (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:19.496 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-dateutil. 2020-11-19T02:49:19.512 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-19T02:49:19.554 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-19T02:49:19.899 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-19T02:49:19.912 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-19T02:49:19.949 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-19T02:49:19.966 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-dateutil. 2020-11-19T02:49:19.980 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-19T02:49:20.026 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-19T02:49:20.268 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-19T02:49:20.281 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-19T02:49:20.318 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-19T02:49:20.377 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-19T02:49:20.392 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-19T02:49:20.436 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-19T02:49:20.662 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cryptography. 2020-11-19T02:49:21.235 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-19T02:49:21.236 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-19T02:49:21.236 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-19T02:49:21.237 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cryptography. 2020-11-19T02:49:21.237 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-19T02:49:21.237 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-19T02:49:21.239 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-19T02:49:21.240 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-19T02:49:21.241 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-openssl. 2020-11-19T02:49:21.243 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-19T02:49:21.243 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-19T02:49:21.459 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-19T02:49:21.477 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:21.511 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-openssl. 2020-11-19T02:49:21.518 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:21.527 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-19T02:49:21.569 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-19T02:49:21.855 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-19T02:49:21.870 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:21.872 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-19T02:49:21.887 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-19T02:49:21.913 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:21.930 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-19T02:49:22.233 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-19T02:49:22.248 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-19T02:49:22.275 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-19T02:49:22.291 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-19T02:49:22.308 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-19T02:49:22.334 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-19T02:49:22.646 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-19T02:49:22.660 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-19T02:49:22.661 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jwt. 2020-11-19T02:49:22.678 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-19T02:49:22.704 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-19T02:49:22.720 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-19T02:49:23.014 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-logutils. 2020-11-19T02:49:23.030 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-19T02:49:23.032 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jwt. 2020-11-19T02:49:23.045 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-19T02:49:23.072 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-19T02:49:23.082 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-19T02:49:23.367 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-19T02:49:23.381 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-19T02:49:23.393 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-logutils. 2020-11-19T02:49:23.407 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-19T02:49:23.442 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-19T02:49:23.468 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-19T02:49:23.745 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-mako. 2020-11-19T02:49:23.758 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-19T02:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-19T02:49:23.802 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-19T02:49:23.811 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-19T02:49:23.854 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-19T02:49:24.147 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-19T02:49:24.148 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-mako. 2020-11-19T02:49:24.161 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-19T02:49:24.163 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-19T02:49:24.197 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-19T02:49:24.206 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-19T02:49:24.516 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-19T02:49:24.517 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-19T02:49:24.533 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-19T02:49:24.534 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-19T02:49:24.575 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-19T02:49:24.576 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-19T02:49:24.878 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-webob. 2020-11-19T02:49:24.887 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-19T02:49:24.892 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-19T02:49:24.900 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-19T02:49:24.935 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-19T02:49:24.937 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-19T02:49:25.240 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-webob. 2020-11-19T02:49:25.255 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-19T02:49:25.263 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-bs4. 2020-11-19T02:49:25.277 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-19T02:49:25.298 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-19T02:49:25.322 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-19T02:49:25.675 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-waitress. 2020-11-19T02:49:25.689 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-19T02:49:25.709 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-bs4. 2020-11-19T02:49:25.725 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-19T02:49:25.742 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-19T02:49:25.768 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-19T02:49:26.078 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-tempita. 2020-11-19T02:49:26.094 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-19T02:49:26.097 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-waitress. 2020-11-19T02:49:26.112 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-19T02:49:26.144 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-19T02:49:26.156 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-19T02:49:26.447 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-paste. 2020-11-19T02:49:26.462 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-19T02:49:26.484 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-tempita. 2020-11-19T02:49:26.497 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-19T02:49:26.505 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-19T02:49:26.534 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-19T02:49:26.841 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-19T02:49:26.844 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-paste. 2020-11-19T02:49:26.858 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-19T02:49:26.859 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-19T02:49:26.894 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-19T02:49:26.899 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-19T02:49:27.202 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-19T02:49:27.216 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-19T02:49:27.247 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-19T02:49:27.260 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-19T02:49:27.264 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-19T02:49:27.306 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-19T02:49:27.598 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-webtest. 2020-11-19T02:49:27.614 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-19T02:49:27.657 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-19T02:49:27.659 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-19T02:49:27.674 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-19T02:49:27.717 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-19T02:49:28.018 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pecan. 2020-11-19T02:49:28.019 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-webtest. 2020-11-19T02:49:28.033 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-19T02:49:28.034 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-19T02:49:28.076 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-19T02:49:28.078 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-19T02:49:28.414 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pecan. 2020-11-19T02:49:28.428 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-19T02:49:28.430 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-19T02:49:28.444 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-19T02:49:28.475 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-19T02:49:28.489 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-19T02:49:28.810 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-19T02:49:28.826 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-19T02:49:28.842 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr. 2020-11-19T02:49:28.856 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:28.868 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-19T02:49:28.900 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:29.238 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr. 2020-11-19T02:49:29.254 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:29.314 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:29.758 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mon. 2020-11-19T02:49:29.774 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:29.817 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mon (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:30.286 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mon. 2020-11-19T02:49:30.301 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:30.345 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mon (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:31.323 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-osd. 2020-11-19T02:49:31.337 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:31.390 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-osd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:31.497 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-osd. 2020-11-19T02:49:31.510 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:31.553 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-osd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:33.403 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph. 2020-11-19T02:49:33.418 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../048-ceph_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:33.472 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:33.575 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph. 2020-11-19T02:49:33.592 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../048-ceph_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:33.634 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:33.814 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-fuse. 2020-11-19T02:49:33.830 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:33.872 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:33.885 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-fuse. 2020-11-19T02:49:33.899 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:33.944 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:34.276 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mds. 2020-11-19T02:49:34.295 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:34.326 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mds (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:34.356 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mds. 2020-11-19T02:49:34.370 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:34.414 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mds (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:34.828 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package cephadm. 2020-11-19T02:49:34.843 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../051-cephadm_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:34.891 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package cephadm. 2020-11-19T02:49:34.906 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../051-cephadm_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:34.950 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:35.206 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jinja2. 2020-11-19T02:49:35.220 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-19T02:49:35.236 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jinja2. 2020-11-19T02:49:35.252 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-19T02:49:35.265 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-19T02:49:35.295 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-19T02:49:35.582 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-19T02:49:36.945 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-19T02:49:36.946 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:36.946 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:36.946 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-19T02:49:36.946 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-19T02:49:36.947 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-19T02:49:36.947 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-routes. 2020-11-19T02:49:36.947 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-19T02:49:36.947 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-19T02:49:36.947 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-19T02:49:36.948 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:36.948 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:36.948 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:36.948 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:36.948 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-19T02:49:36.949 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-19T02:49:36.949 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-19T02:49:36.949 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-routes. 2020-11-19T02:49:36.949 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-19T02:49:36.949 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-19T02:49:36.950 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-19T02:49:36.950 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:36.950 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:37.613 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-numpy. 2020-11-19T02:49:37.628 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-19T02:49:37.671 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-19T02:49:37.684 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-numpy. 2020-11-19T02:49:37.697 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-19T02:49:37.742 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-19T02:49:38.243 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-decorator. 2020-11-19T02:49:38.259 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-19T02:49:38.302 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-19T02:49:38.322 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-decorator. 2020-11-19T02:49:38.336 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-19T02:49:38.392 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-19T02:49:38.613 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-scipy. 2020-11-19T02:49:38.627 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-19T02:49:38.672 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-19T02:49:38.700 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-scipy. 2020-11-19T02:49:38.716 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-19T02:49:38.759 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-19T02:49:39.938 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-19T02:49:39.954 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-19T02:49:39.996 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-19T02:49:40.100 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-19T02:49:40.118 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-19T02:49:40.167 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-19T02:49:40.442 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-joblib. 2020-11-19T02:49:40.456 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-19T02:49:40.500 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-19T02:49:40.637 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-joblib. 2020-11-19T02:49:41.266 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-sklearn. 2020-11-19T02:49:41.266 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-19T02:49:41.267 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-19T02:49:41.267 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-19T02:49:41.267 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-19T02:49:41.267 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-sklearn. 2020-11-19T02:49:41.268 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-19T02:49:41.268 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-19T02:49:41.467 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-19T02:49:41.476 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:41.518 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:41.646 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-19T02:49:41.660 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:41.704 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:42.595 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-monotonic. 2020-11-19T02:49:42.612 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-19T02:49:42.654 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-19T02:49:42.857 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-monotonic. 2020-11-19T02:49:42.871 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-19T02:49:42.915 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-19T02:49:42.965 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-fasteners. 2020-11-19T02:49:42.980 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-19T02:49:43.023 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-19T02:49:43.201 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-fasteners. 2020-11-19T02:49:43.218 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-19T02:49:43.260 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-19T02:49:43.309 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-httplib2. 2020-11-19T02:49:43.325 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-19T02:49:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-19T02:49:43.562 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-httplib2. 2020-11-19T02:49:43.576 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-19T02:49:43.637 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-19T02:49:43.654 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-19T02:49:43.669 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-19T02:49:43.703 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-19T02:49:43.940 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-19T02:49:43.956 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-19T02:49:43.998 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-19T02:49:44.024 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-19T02:49:44.040 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-19T02:49:44.082 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-19T02:49:44.368 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-19T02:49:44.377 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rsa. 2020-11-19T02:49:44.385 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-19T02:49:44.393 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-19T02:49:44.426 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-19T02:49:44.459 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-19T02:49:44.771 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-19T02:49:44.786 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-19T02:49:44.829 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-19T02:49:44.862 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rsa. 2020-11-19T02:49:44.877 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-19T02:49:44.921 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-19T02:49:45.241 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-19T02:49:45.256 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-19T02:49:45.292 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-19T02:49:45.299 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-19T02:49:45.305 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-19T02:49:45.350 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-19T02:49:45.719 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-setuptools. 2020-11-19T02:49:46.538 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-19T02:49:46.538 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-19T02:49:46.539 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-19T02:49:46.539 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-setuptools. 2020-11-19T02:49:46.539 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-19T02:49:46.540 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-19T02:49:46.540 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-19T02:49:46.541 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-19T02:49:46.617 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-websocket. 2020-11-19T02:49:46.631 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-websocket. 2020-11-19T02:49:46.633 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-19T02:49:46.645 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-19T02:49:46.676 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-19T02:49:46.689 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-19T02:49:47.004 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-19T02:49:47.019 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-19T02:49:47.042 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-19T02:49:47.056 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-19T02:49:47.062 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-19T02:49:47.109 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-19T02:49:47.499 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-19T02:49:47.515 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:47.529 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-19T02:49:47.544 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:47.557 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:47.588 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:47.885 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-19T02:49:47.900 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-19T02:49:47.916 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-19T02:49:47.931 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-19T02:49:47.943 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-19T02:49:47.974 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-19T02:49:48.363 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-19T02:49:48.377 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-19T02:49:48.385 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-19T02:49:48.402 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-19T02:49:48.425 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-19T02:49:48.480 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-19T02:49:48.733 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-19T02:49:48.748 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:48.779 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-19T02:49:48.791 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-rook (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:48.793 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T02:49:48.838 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-rook (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:49.174 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-19T02:49:49.177 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-19T02:49:49.188 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-19T02:49:49.192 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-19T02:49:49.233 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-19T02:49:49.235 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-19T02:49:49.585 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-19T02:49:49.588 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-19T02:49:49.600 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-19T02:49:49.603 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-19T02:49:49.721 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-19T02:49:49.724 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-19T02:49:49.990 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package jq. 2020-11-19T02:49:50.001 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package jq. 2020-11-19T02:49:50.008 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-19T02:49:50.015 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-19T02:49:50.048 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-19T02:49:50.060 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-19T02:49:50.301 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package socat. 2020-11-19T02:49:50.316 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-19T02:49:50.337 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package socat. 2020-11-19T02:49:50.351 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-19T02:49:50.368 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-19T02:49:50.398 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-19T02:49:50.763 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-19T02:49:51.500 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-19T02:49:51.500 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-19T02:49:51.500 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-19T02:49:51.501 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package xmlstarlet. 2020-11-19T02:49:51.501 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-19T02:49:51.501 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-19T02:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-19T02:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-19T02:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package xmlstarlet. 2020-11-19T02:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-19T02:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-19T02:49:51.537 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-test. 2020-11-19T02:49:51.547 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-test. 2020-11-19T02:49:51.551 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:51.562 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:51.595 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-test (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:51.606 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-test (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:55.226 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-19T02:49:55.240 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:55.283 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libcephfs-dev (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:55.314 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-19T02:49:55.324 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:49:55.368 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libcephfs-dev (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:49:55.543 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package nvme-cli. 2020-11-19T02:49:55.557 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-19T02:49:56.156 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package nvme-cli. 2020-11-19T02:49:56.156 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-19T02:49:56.157 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-19T02:49:56.157 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-attr. 2020-11-19T02:49:56.157 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-19T02:49:56.157 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-19T02:49:56.158 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-19T02:49:56.158 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-attr. 2020-11-19T02:49:56.158 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-19T02:49:56.158 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-19T02:49:56.350 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-funcsigs. 2020-11-19T02:49:56.364 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-19T02:49:56.386 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-funcsigs. 2020-11-19T02:49:56.400 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-19T02:49:56.408 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-19T02:49:56.436 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-19T02:49:56.697 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-pluggy. 2020-11-19T02:49:56.711 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-pluggy. 2020-11-19T02:49:56.713 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-19T02:49:56.731 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-19T02:49:56.755 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-19T02:49:56.769 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-19T02:49:57.072 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-py. 2020-11-19T02:49:57.087 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-19T02:49:57.091 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-py. 2020-11-19T02:49:57.105 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-19T02:49:57.130 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-19T02:49:57.149 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-19T02:49:57.467 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-pytest. 2020-11-19T02:49:57.477 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-pytest. 2020-11-19T02:49:57.483 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-19T02:49:57.492 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-19T02:49:57.533 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-19T02:49:57.535 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-19T02:49:57.880 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-attr. 2020-11-19T02:49:57.895 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-attr. 2020-11-19T02:49:57.895 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-19T02:49:57.909 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-19T02:49:57.938 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-19T02:49:57.961 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-19T02:49:58.250 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cycler. 2020-11-19T02:49:58.264 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-19T02:49:58.272 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cycler. 2020-11-19T02:49:58.289 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-19T02:49:58.308 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-19T02:49:58.330 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-19T02:49:58.569 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-webencodings. 2020-11-19T02:49:58.582 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-19T02:49:58.610 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-19T02:49:58.617 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-webencodings. 2020-11-19T02:49:58.633 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-19T02:49:58.675 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-19T02:49:58.897 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-html5lib. 2020-11-19T02:49:58.913 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-19T02:49:58.955 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-19T02:49:58.978 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-html5lib. 2020-11-19T02:49:58.991 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-19T02:49:59.036 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-19T02:49:59.316 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-19T02:49:59.331 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-19T02:49:59.374 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-19T02:49:59.381 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-19T02:49:59.396 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-19T02:49:59.440 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-19T02:49:59.811 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-19T02:49:59.828 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-19T02:49:59.869 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-19T02:49:59.918 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-19T02:49:59.932 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-19T02:49:59.985 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-19T02:50:00.304 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-tz. 2020-11-19T02:50:00.319 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-19T02:50:00.362 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-19T02:50:00.647 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-tz. 2020-11-19T02:50:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-19T02:50:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-19T02:50:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-19T02:50:01.153 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-19T02:50:01.153 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-19T02:50:01.153 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-19T02:50:01.154 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-19T02:50:01.154 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-19T02:50:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-olefile. 2020-11-19T02:50:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-19T02:50:01.500 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-19T02:50:01.756 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-olefile. 2020-11-19T02:50:01.772 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-19T02:50:01.815 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-19T02:50:01.840 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pastescript. 2020-11-19T02:50:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-19T02:50:01.907 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-19T02:50:02.125 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pastescript. 2020-11-19T02:50:02.142 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-19T02:50:02.183 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-19T02:50:02.285 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-19T02:50:02.300 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-19T02:50:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-19T02:50:02.537 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-19T02:50:02.554 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-19T02:50:02.595 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-19T02:50:02.688 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pluggy. 2020-11-19T02:50:02.705 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-19T02:50:02.746 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-19T02:50:02.968 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pluggy. 2020-11-19T02:50:02.982 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-19T02:50:03.023 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-19T02:50:03.074 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-py. 2020-11-19T02:50:03.088 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-19T02:50:03.132 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-19T02:50:03.326 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-py. 2020-11-19T02:50:03.341 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-19T02:50:03.384 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-19T02:50:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-19T02:50:03.483 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-19T02:50:03.527 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-19T02:50:03.745 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-19T02:50:03.762 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-19T02:50:03.804 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-19T02:50:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pytest. 2020-11-19T02:50:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-19T02:50:03.904 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-19T02:50:04.140 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pytest. 2020-11-19T02:50:04.156 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-19T02:50:04.198 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-19T02:50:04.258 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-simplejson. 2020-11-19T02:50:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-19T02:50:04.316 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-19T02:50:04.593 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-simplejson. 2020-11-19T02:50:04.610 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-19T02:50:04.644 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-19T02:50:04.651 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-19T02:50:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-19T02:50:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-19T02:50:04.971 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-19T02:50:04.985 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-19T02:50:05.029 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-19T02:50:05.164 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package radosgw. 2020-11-19T02:50:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../110-radosgw_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:50:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking radosgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:05.501 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package radosgw. 2020-11-19T02:50:05.514 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../110-radosgw_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:50:05.558 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking radosgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:06.253 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package rbd-fuse. 2020-11-19T02:50:06.267 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:50:06.311 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking rbd-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:06.578 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package rbd-fuse. 2020-11-19T02:50:06.592 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T02:50:06.616 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-19T02:50:06.636 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking rbd-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:06.933 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-19T02:50:06.972 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-19T02:50:07.299 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-19T02:50:07.301 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-19T02:50:07.427 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-19T02:50:07.667 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-19T02:50:07.793 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-19T02:50:08.054 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-19T02:50:08.429 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-19T02:50:08.439 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-19T02:50:08.732 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-19T02:50:08.813 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-19T02:50:09.033 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-19T02:50:09.100 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-19T02:50:09.235 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-19T02:50:09.360 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-19T02:50:09.409 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-19T02:50:09.610 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-19T02:50:09.687 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-19T02:50:09.736 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-19T02:50:10.083 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-19T02:50:10.096 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-19T02:50:10.381 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-19T02:50:10.459 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-19T02:50:10.699 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-ceph-argparse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:11.759 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-19T02:50:11.759 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-ceph-argparse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:11.759 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-19T02:50:11.760 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-19T02:50:11.760 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-19T02:50:11.760 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-19T02:50:11.760 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-19T02:50:11.795 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-19T02:50:12.372 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-19T02:50:12.707 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-19T02:50:12.731 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-py (1.5.2-1) ... 2020-11-19T02:50:13.049 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-19T02:50:13.226 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-19T02:50:13.384 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-py (1.5.2-1) ... 2020-11-19T02:50:13.561 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-19T02:50:13.610 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-19T02:50:13.651 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-19T02:50:13.896 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-19T02:50:13.936 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-19T02:50:14.022 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-19T02:50:14.035 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-19T02:50:14.215 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-19T02:50:14.254 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-19T02:50:14.380 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-19T02:50:14.532 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-19T02:50:14.589 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-19T02:50:14.850 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-19T02:50:14.891 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-19T02:50:15.151 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-19T02:50:15.210 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-19T02:50:15.510 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-19T02:50:15.523 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-19T02:50:15.862 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-19T02:50:16.157 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-19T02:50:16.165 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:16.229 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-19T02:50:16.522 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:17.147 INFO:teuthology.orchestra.run.smithi160.stdout:Adding system user cephadm....done 2020-11-19T02:50:17.311 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-19T02:50:17.340 INFO:teuthology.orchestra.run.smithi086.stdout:Adding system user cephadm....done 2020-11-19T02:50:17.495 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-19T02:50:17.604 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-19T02:50:17.730 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-19T02:50:17.813 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-19T02:50:17.872 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-19T02:50:17.939 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-19T02:50:18.065 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-19T02:50:18.198 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-19T02:50:18.425 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-19T02:50:18.475 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-19T02:50:18.584 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-19T02:50:18.726 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-19T02:50:18.852 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-19T02:50:18.969 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:19.195 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:19.287 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-19T02:50:19.522 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-19T02:50:19.563 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-19T02:50:19.689 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-19T02:50:19.823 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-19T02:50:19.949 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-19T02:50:20.056 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-19T02:50:20.308 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-19T02:50:20.400 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-19T02:50:20.472 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-19T02:50:20.513 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-19T02:50:20.669 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-19T02:50:20.752 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-19T02:50:20.760 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-19T02:50:20.792 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-19T02:50:21.038 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-19T02:50:21.045 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-19T02:50:21.355 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-19T02:50:21.370 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-19T02:50:21.664 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-19T02:50:21.721 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-19T02:50:21.999 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-19T02:50:22.031 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-19T02:50:22.240 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-19T02:50:22.316 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-19T02:50:22.518 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-19T02:50:22.617 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-19T02:50:22.690 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-19T02:50:22.919 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-19T02:50:22.943 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-19T02:50:23.010 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-19T02:50:23.119 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-19T02:50:23.246 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-19T02:50:23.272 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-19T02:50:23.447 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-19T02:50:23.513 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-19T02:50:23.573 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-19T02:50:23.639 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-19T02:50:23.858 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-19T02:50:24.001 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-19T02:50:24.041 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-19T02:50:24.150 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-19T02:50:24.411 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-19T02:50:24.427 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-19T02:50:24.537 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-19T02:50:24.744 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-19T02:50:24.838 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-19T02:50:25.045 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-19T02:50:25.139 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-19T02:50:25.346 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-19T02:50:25.450 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-19T02:50:25.723 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-19T02:50:27.065 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-19T02:50:27.066 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-19T02:50:27.066 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-19T02:50:27.066 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-19T02:50:27.067 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-19T02:50:27.067 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-19T02:50:27.067 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-19T02:50:27.067 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-19T02:50:27.105 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-19T02:50:27.345 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-19T02:50:27.471 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-19T02:50:27.473 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-19T02:50:27.799 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-19T02:50:27.950 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-19T02:50:28.637 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-19T02:50:28.969 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-19T02:50:29.097 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-19T02:50:29.287 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-19T02:50:29.413 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-19T02:50:29.426 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-19T02:50:29.750 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-19T02:50:29.781 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-19T02:50:29.885 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-19T02:50:30.040 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-19T02:50:30.166 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-19T02:50:30.253 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-19T02:50:30.267 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librados2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:30.385 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-19T02:50:30.511 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-19T02:50:30.631 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-19T02:50:30.696 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libcephfs2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:31.922 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librados2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:31.923 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-19T02:50:31.923 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libcephfs2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:31.924 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libcephfs-dev (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:31.924 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rados (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:31.925 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-19T02:50:31.925 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libcephfs-dev (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:31.926 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rados (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:31.926 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-19T02:50:31.926 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-19T02:50:31.955 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-19T02:50:32.133 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-19T02:50:32.272 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-19T02:50:32.442 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-19T02:50:32.599 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-19T02:50:33.388 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-19T02:50:33.520 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-19T02:50:34.234 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-19T02:50:34.358 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-19T02:50:34.535 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libradosstriper1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:34.667 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libradosstriper1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:34.678 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cephfs (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:34.810 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cephfs (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:35.013 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librgw2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:35.103 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librgw2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:35.142 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:35.229 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:35.355 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-19T02:50:35.355 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-19T02:50:35.439 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-19T02:50:35.439 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-19T02:50:35.615 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librbd1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:35.706 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librbd1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:35.732 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-19T02:50:35.805 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-19T02:50:35.831 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-19T02:50:35.854 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-19T02:50:35.921 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-19T02:50:35.962 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-19T02:50:36.169 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-19T02:50:36.276 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-19T02:50:36.846 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-modules-core (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:36.913 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-modules-core (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:36.989 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-19T02:50:37.038 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-19T02:50:37.315 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:37.381 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:37.424 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up rbd-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:37.507 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up rbd-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:37.525 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rbd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:37.626 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:37.633 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rbd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:37.768 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:38.160 INFO:teuthology.orchestra.run.smithi160.stdout:Adding group ceph....done 2020-11-19T02:50:38.294 INFO:teuthology.orchestra.run.smithi086.stdout:Adding group ceph....done 2020-11-19T02:50:38.546 INFO:teuthology.orchestra.run.smithi160.stdout:Adding system user ceph....done 2020-11-19T02:50:38.631 INFO:teuthology.orchestra.run.smithi086.stdout:Adding system user ceph....done 2020-11-19T02:50:38.722 INFO:teuthology.orchestra.run.smithi160.stdout:Setting system user ceph properties....done 2020-11-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi086.stdout:Setting system user ceph properties....done 2020-11-19T02:50:38.879 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-19T02:50:39.010 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-19T02:50:39.085 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-19T02:50:39.228 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-19T02:50:39.344 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up radosgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:39.528 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up radosgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:39.602 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-19T02:50:39.602 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-19T02:50:39.794 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-19T02:50:39.794 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-19T02:50:39.997 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-test (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:40.122 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-base (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:40.214 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-test (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:40.332 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-base (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:40.467 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-19T02:50:40.630 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-19T02:50:40.716 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:41.230 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:41.231 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-19T02:50:41.231 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-19T02:50:41.232 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-19T02:50:41.232 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-19T02:50:41.293 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:41.377 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:41.445 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-osd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:41.519 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-osd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:41.897 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-19T02:50:41.898 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-19T02:50:41.963 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-19T02:50:41.964 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-19T02:50:42.146 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:42.229 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:42.289 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-rook (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:42.364 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-rook (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:42.432 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mds (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:42.658 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mds (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:42.679 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-19T02:50:42.680 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-19T02:50:42.862 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-19T02:50:42.863 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-19T02:50:42.933 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mon (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:43.129 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-19T02:50:43.129 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-19T02:50:43.134 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mon (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:43.353 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-19T02:50:43.353 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-19T02:50:43.401 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:43.502 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-dashboard (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:43.635 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:43.636 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:43.761 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-dashboard (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:43.762 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-19T02:50:43.912 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T02:50:43.955 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-19T02:50:44.039 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-19T02:50:44.249 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-19T02:50:46.041 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-19T02:50:46.401 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-19T02:50:46.402 INFO:teuthology.orchestra.run.smithi086.stdout:ureadahead will be reprofiled on next reboot 2020-11-19T02:50:46.402 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-19T02:50:46.402 INFO:teuthology.orchestra.run.smithi160.stdout:ureadahead will be reprofiled on next reboot 2020-11-19T02:50:46.403 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-19T02:50:46.691 INFO:teuthology.orchestra.run.smithi160.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-19T02:50:46.693 DEBUG:teuthology.parallel:result is None 2020-11-19T02:50:46.787 INFO:teuthology.orchestra.run.smithi086.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-19T02:50:46.789 DEBUG:teuthology.parallel:result is None 2020-11-19T02:50:46.789 INFO:teuthology.orchestra.run.smithi086:> sudo lsb_release -is 2020-11-19T02:50:46.838 INFO:teuthology.orchestra.run.smithi086.stdout:Ubuntu 2020-11-19T02:50:46.843 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T02:50:46.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T02:50:47.020 INFO:teuthology.orchestra.run.smithi086:> dpkg-query -W -f '${Version}' ceph 2020-11-19T02:50:47.046 INFO:teuthology.orchestra.run.smithi086.stdout:16.0.0-7199-g409eac50-1bionic 2020-11-19T02:50:47.047 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7199-g409eac50-1bionic 2020-11-19T02:50:47.047 INFO:teuthology.task.install:The correct ceph version 16.0.0-7199-g409eac50-1bionic is installed. 2020-11-19T02:50:47.048 INFO:teuthology.orchestra.run.smithi160:> sudo lsb_release -is 2020-11-19T02:50:47.103 INFO:teuthology.orchestra.run.smithi160.stdout:Ubuntu 2020-11-19T02:50:47.108 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T02:50:47.109 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T02:50:47.274 INFO:teuthology.orchestra.run.smithi160:> dpkg-query -W -f '${Version}' ceph 2020-11-19T02:50:47.299 INFO:teuthology.orchestra.run.smithi160.stdout:16.0.0-7199-g409eac50-1bionic 2020-11-19T02:50:47.300 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7199-g409eac50-1bionic 2020-11-19T02:50:47.300 INFO:teuthology.task.install:The correct ceph version 16.0.0-7199-g409eac50-1bionic is installed. 2020-11-19T02:50:47.301 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-19T02:50:47.301 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:47.302 INFO:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-19T02:50:47.307 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-19T02:50:47.317 INFO:teuthology.orchestra.run.smithi086.stderr:24+1 records in 2020-11-19T02:50:47.317 INFO:teuthology.orchestra.run.smithi086.stderr:24+1 records out 2020-11-19T02:50:47.318 INFO:teuthology.orchestra.run.smithi086.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.00022062 s, 57.4 MB/s 2020-11-19T02:50:47.319 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:50:47.320 INFO:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-19T02:50:47.346 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-19T02:50:47.355 INFO:teuthology.orchestra.run.smithi160.stderr:24+1 records in 2020-11-19T02:50:47.355 INFO:teuthology.orchestra.run.smithi160.stderr:24+1 records out 2020-11-19T02:50:47.355 INFO:teuthology.orchestra.run.smithi160.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000162697 s, 77.9 MB/s 2020-11-19T02:50:47.356 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-19T02:50:47.357 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:47.357 INFO:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2020-11-19T02:50:47.364 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-19T02:50:47.373 INFO:teuthology.orchestra.run.smithi086.stderr:5+1 records in 2020-11-19T02:50:47.373 INFO:teuthology.orchestra.run.smithi086.stderr:5+1 records out 2020-11-19T02:50:47.373 INFO:teuthology.orchestra.run.smithi086.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 0.000109233 s, 28.0 MB/s 2020-11-19T02:50:47.375 INFO:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-19T02:50:47.431 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:50:47.431 INFO:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2020-11-19T02:50:47.436 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-19T02:50:47.445 INFO:teuthology.orchestra.run.smithi160.stderr:5+1 records in 2020-11-19T02:50:47.446 INFO:teuthology.orchestra.run.smithi160.stderr:5+1 records out 2020-11-19T02:50:47.446 INFO:teuthology.orchestra.run.smithi160.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.8887e-05 s, 30.9 MB/s 2020-11-19T02:50:47.447 INFO:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-19T02:50:47.506 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-19T02:50:47.506 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:47.507 INFO:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-19T02:50:47.512 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-19T02:50:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:299 bytes copied, 7.3156e-05 s, 4.1 MB/s 2020-11-19T02:50:47.523 INFO:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-19T02:50:47.582 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:50:47.583 INFO:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-19T02:50:47.588 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-19T02:50:47.596 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:50:47.597 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:50:47.597 INFO:teuthology.orchestra.run.smithi160.stderr:299 bytes copied, 6.546e-05 s, 4.6 MB/s 2020-11-19T02:50:47.598 INFO:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-19T02:50:47.655 INFO:teuthology.run_tasks:Running task ceph... 2020-11-19T02:50:47.724 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-19T02:50:47.725 INFO:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2020-11-19T02:50:47.729 INFO:teuthology.orchestra.run.smithi160:> sudo chmod 777 /var/log/ceph 2020-11-19T02:50:47.743 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-19T02:50:47.743 INFO:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-19T02:50:47.786 INFO:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-19T02:50:47.803 INFO:tasks.ceph:Creating extra log directories... 2020-11-19T02:50:47.804 INFO:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-19T02:50:47.846 INFO:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-19T02:50:47.890 INFO:tasks.ceph:Creating ceph cluster c1... 2020-11-19T02:50:47.890 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-19T02:50:47.891 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633829', 'branch': 'wip-yuval-fix-47904', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5633829', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuval-fix-47904'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuval-fix-47904', 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuval-fix-47904', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.6672', 'targets': {'smithi086.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOX2yzewpn2i4e07+3TrL99b0P2l1x7R1S9+FHleR/gamdR1+xV0/Rm+YdgPbTxuw+O/rQODT7fLlVl6UBihJS2pfoS+X9X0o1znO23OYTzkehepKz6xqNXkHKnxEdAvddqlisgyrYnC5j6KNiWIlSybv25VJ//uZQqrkKU74Uz2fp6l83hoQjttdrZoowG1KmXEz5gdqczArtjOO7oL81jf7RAY5Abrs0XdnVq0ifxD6lSpsxlv3HiEO5/E4FhEErv5xcawzDG8Nle9rjLjnO9QNp6H87gLyfTj72zE0fgSHbB5F08vO/AxZhZynig/FWjffG2ftCmO9tOywwHVz9', 'smithi160.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDwLcfm2rpI0BEMcouJXOuPRNnfF7ta3SKliynVLHNZbLhzUxe85D2ZtcpDnU2EwXhCEv37c+wXgbEu92FsekOyP246U5k4aX74T8LiGLHuvCXgyvqwq9qjWsDFS4oYn6yuCUzv9aVMpkX6K8eq7vmknRVFEA3Xf3hWWIUeltVOeapmEutN6AlNm+dRfX8rNqiYubr8cWyr5nHsDx6uhjTJBnu1HDghlbttSZNPzeOy95IRTqQUXJHk0G+zBjG5wf+FaeeReGIj/ttMqUQsyHgT5U+zjyGJm0fCTL7ZQF4Rh0e0x+Bj4cq1FLcvNUSXUHmH8q8o0B40fXWovLgRPNbh'}} 2020-11-19T02:50:47.892 INFO:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-19T02:50:47.902 INFO:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-19T02:50:47.912 INFO:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2020-11-19T02:50:47.953 INFO:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/run/ceph 2020-11-19T02:50:47.969 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:47.970 INFO:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2020-11-19T02:50:48.009 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-19T02:50:48.010 INFO:teuthology.orchestra.run.smithi086.stdout:/dev/vg_nvme/lv_1 2020-11-19T02:50:48.010 INFO:teuthology.orchestra.run.smithi086.stdout:/dev/vg_nvme/lv_2 2020-11-19T02:50:48.010 INFO:teuthology.orchestra.run.smithi086.stdout:/dev/vg_nvme/lv_3 2020-11-19T02:50:48.011 INFO:teuthology.orchestra.run.smithi086.stdout:/dev/vg_nvme/lv_4 2020-11-19T02:50:48.011 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:48.011 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:48.011 INFO:teuthology.orchestra.run.smithi086.stderr:72 bytes copied, 0.000151791 s, 474 kB/s 2020-11-19T02:50:48.012 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-19T02:50:48.012 INFO:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-19T02:50:48.059 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-19T02:50:48.059 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T02:50:48.060 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 342 Links: 1 2020-11-19T02:50:48.060 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T02:50:48.061 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2020-11-19 02:47:56.870114056 +0000 2020-11-19T02:50:48.061 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2020-11-19 02:47:56.766166058 +0000 2020-11-19T02:50:48.061 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2020-11-19 02:47:56.766166058 +0000 2020-11-19T02:50:48.062 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2020-11-19T02:50:48.069 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2020-11-19T02:50:48.069 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2020-11-19T02:50:48.070 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000330414 s, 1.5 MB/s 2020-11-19T02:50:48.073 INFO:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-19T02:50:48.127 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-19T02:50:48.127 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T02:50:48.128 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-11-19T02:50:48.128 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T02:50:48.128 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2020-11-19 02:47:57.086006051 +0000 2020-11-19T02:50:48.128 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2020-11-19 02:47:56.982058054 +0000 2020-11-19T02:50:48.128 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2020-11-19 02:47:56.982058054 +0000 2020-11-19T02:50:48.129 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2020-11-19T02:50:48.136 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2020-11-19T02:50:48.136 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2020-11-19T02:50:48.136 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000295063 s, 1.7 MB/s 2020-11-19T02:50:48.139 INFO:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-19T02:50:48.190 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-19T02:50:48.191 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T02:50:48.192 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-11-19T02:50:48.192 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T02:50:48.193 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2020-11-19 02:47:57.313892046 +0000 2020-11-19T02:50:48.194 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2020-11-19 02:47:57.201948049 +0000 2020-11-19T02:50:48.194 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2020-11-19 02:47:57.201948049 +0000 2020-11-19T02:50:48.195 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2020-11-19T02:50:48.199 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2020-11-19T02:50:48.200 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2020-11-19T02:50:48.200 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000328531 s, 1.6 MB/s 2020-11-19T02:50:48.207 INFO:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-19T02:50:48.261 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T02:50:48.261 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-11-19T02:50:48.262 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T02:50:48.262 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2020-11-19 02:47:57.521788042 +0000 2020-11-19T02:50:48.263 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2020-11-19 02:47:57.425836044 +0000 2020-11-19T02:50:48.263 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2020-11-19 02:47:57.425836044 +0000 2020-11-19T02:50:48.264 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2020-11-19T02:50:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2020-11-19T02:50:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2020-11-19T02:50:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000400404 s, 1.3 MB/s 2020-11-19T02:50:48.275 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T02:50:48.276 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-19T02:50:48.276 INFO:tasks.ceph:Generating config... 2020-11-19T02:50:48.280 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-19T02:50:48.280 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-19T02:50:48.281 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-19T02:50:48.282 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T02:50:48.282 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-19T02:50:48.283 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-19T02:50:48.283 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-19T02:50:48.283 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-19T02:50:48.284 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-19T02:50:48.284 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-19T02:50:48.284 INFO:tasks.ceph:[client] setuser = ceph 2020-11-19T02:50:48.285 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-19T02:50:48.285 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-19T02:50:48.285 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-19T02:50:48.285 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-19T02:50:48.286 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-19T02:50:48.286 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-19T02:50:48.286 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-19T02:50:48.287 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-19T02:50:48.287 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-19T02:50:48.287 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-19T02:50:48.287 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-19T02:50:48.288 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-19T02:50:48.288 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-19T02:50:48.288 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-19T02:50:48.289 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-19T02:50:48.289 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-19T02:50:48.289 INFO:tasks.ceph:Setting up c1.mon.a... 2020-11-19T02:50:48.290 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-11-19T02:50:48.359 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/c1.keyring 2020-11-19T02:50:48.361 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-11-19T02:50:48.444 INFO:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T02:50:48.507 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.86')] 2020-11-19T02:50:48.507 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.86', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-19T02:50:48.508 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:48.509 INFO:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T02:50:48.563 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T02:50:48.567 INFO:teuthology.orchestra.run.smithi086.stderr:6+1 records in 2020-11-19T02:50:48.567 INFO:teuthology.orchestra.run.smithi086.stderr:6+1 records out 2020-11-19T02:50:48.568 INFO:teuthology.orchestra.run.smithi086.stderr:3135 bytes (3.1 kB, 3.1 KiB) copied, 0.00228394 s, 1.4 MB/s 2020-11-19T02:50:48.568 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.86 --print /home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:48.643 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:50:48.644 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:50:48.644 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:48.645 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: generated fsid b27f53c1-7dd2-4d8a-83e5-41e60c239cf7 2020-11-19T02:50:48.645 INFO:teuthology.orchestra.run.smithi086.stdout:epoch 0 2020-11-19T02:50:48.645 INFO:teuthology.orchestra.run.smithi086.stdout:fsid b27f53c1-7dd2-4d8a-83e5-41e60c239cf7 2020-11-19T02:50:48.645 INFO:teuthology.orchestra.run.smithi086.stdout:last_changed 2020-11-19T02:50:48.642772+0000 2020-11-19T02:50:48.647 INFO:teuthology.orchestra.run.smithi086.stdout:created 2020-11-19T02:50:48.642772+0000 2020-11-19T02:50:48.647 INFO:teuthology.orchestra.run.smithi086.stdout:min_mon_release 0 (unknown) 2020-11-19T02:50:48.647 INFO:teuthology.orchestra.run.smithi086.stdout:election_strategy: 1 2020-11-19T02:50:48.648 INFO:teuthology.orchestra.run.smithi086.stdout:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2020-11-19T02:50:48.648 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-11-19T02:50:48.650 INFO:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T02:50:48.699 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID b27f53c1-7dd2-4d8a-83e5-41e60c239cf7... 2020-11-19T02:50:48.700 INFO:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-19T02:50:48.742 INFO:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-19T02:50:48.774 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2020-11-19T02:50:48.775 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2020-11-19T02:50:48.775 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-19T02:50:48.775 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2020-11-19T02:50:48.776 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.776 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2020-11-19T02:50:48.776 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.776 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2020-11-19T02:50:48.776 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.777 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2020-11-19T02:50:48.777 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2020-11-19T02:50:48.777 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.777 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2020-11-19T02:50:48.777 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2020-11-19T02:50:48.778 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.778 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-19T02:50:48.778 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2020-11-19T02:50:48.778 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.779 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2020-11-19T02:50:48.779 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-19T02:50:48.779 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.779 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2020-11-19T02:50:48.780 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.780 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2020-11-19T02:50:48.780 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2020-11-19T02:50:48.780 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2020-11-19T02:50:48.781 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2020-11-19T02:50:48.781 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2020-11-19T02:50:48.781 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2020-11-19T02:50:48.781 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2020-11-19T02:50:48.782 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-19T02:50:48.782 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-19T02:50:48.782 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2020-11-19T02:50:48.782 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.783 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-19T02:50:48.783 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.783 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2020-11-19T02:50:48.784 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.784 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2020-11-19T02:50:48.784 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.784 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2020-11-19T02:50:48.784 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2020-11-19T02:50:48.785 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2020-11-19T02:50:48.785 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = 172.21.15.86 2020-11-19T02:50:48.785 INFO:teuthology.orchestra.run.smithi086.stdout: osd heartbeat grace = 40 2020-11-19T02:50:48.785 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = b27f53c1-7dd2-4d8a-83e5-41e60c239cf7 2020-11-19T02:50:48.786 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.786 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2020-11-19T02:50:48.786 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2020-11-19T02:50:48.786 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.786 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2020-11-19T02:50:48.787 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2020-11-19T02:50:48.787 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.787 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2020-11-19T02:50:48.787 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2020-11-19T02:50:48.788 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.788 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2020-11-19T02:50:48.788 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2020-11-19T02:50:48.788 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2020-11-19T02:50:48.789 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.789 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2020-11-19T02:50:48.789 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2020-11-19T02:50:48.789 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.789 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2020-11-19T02:50:48.790 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.790 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2020-11-19T02:50:48.790 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.790 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2020-11-19T02:50:48.791 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.791 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2020-11-19T02:50:48.791 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2020-11-19T02:50:48.791 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.792 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2020-11-19T02:50:48.792 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2020-11-19T02:50:48.792 INFO:teuthology.orchestra.run.smithi086.stdout: debug filestore = 20 2020-11-19T02:50:48.792 INFO:teuthology.orchestra.run.smithi086.stdout: debug journal = 20 2020-11-19T02:50:48.792 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 20 2020-11-19T02:50:48.793 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 25 2020-11-19T02:50:48.793 INFO:teuthology.orchestra.run.smithi086.stdout: osd fast shutdown = False 2020-11-19T02:50:48.793 INFO:teuthology.orchestra.run.smithi086.stdout: rocksdb delete range threshold = 0 2020-11-19T02:50:48.793 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.794 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2020-11-19T02:50:48.794 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2020-11-19T02:50:48.794 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2020-11-19T02:50:48.794 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2020-11-19T02:50:48.795 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2020-11-19T02:50:48.795 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2020-11-19T02:50:48.795 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2020-11-19T02:50:48.795 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2020-11-19T02:50:48.796 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.796 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2020-11-19T02:50:48.796 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2020-11-19T02:50:48.796 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2020-11-19T02:50:48.796 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2020-11-19T02:50:48.797 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2020-11-19T02:50:48.797 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2020-11-19T02:50:48.797 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2020-11-19T02:50:48.797 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2020-11-19T02:50:48.798 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2020-11-19T02:50:48.798 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2020-11-19T02:50:48.798 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2020-11-19T02:50:48.798 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd crush smoke test = False 2020-11-19T02:50:48.798 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:50:48.799 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2020-11-19T02:50:48.799 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2020-11-19T02:50:48.799 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2020-11-19T02:50:48.799 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2020-11-19T02:50:48.800 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-19T02:50:48.800 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-19T02:50:48.800 INFO:teuthology.orchestra.run.smithi086.stdout: debug rgw = 20 2020-11-19T02:50:48.800 INFO:teuthology.orchestra.run.smithi086.stdout: rgw crypt require ssl = False 2020-11-19T02:50:48.801 INFO:teuthology.orchestra.run.smithi086.stdout: rgw crypt s3 kms backend = testing 2020-11-19T02:50:48.801 INFO:teuthology.orchestra.run.smithi086.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T02:50:48.801 INFO:teuthology.orchestra.run.smithi086.stdout: rgw curl low speed time = 300 2020-11-19T02:50:48.801 INFO:teuthology.orchestra.run.smithi086.stdout: rgw data log num shards = 4 2020-11-19T02:50:48.801 INFO:teuthology.orchestra.run.smithi086.stdout: rgw md log max shards = 4 2020-11-19T02:50:48.802 INFO:teuthology.orchestra.run.smithi086.stdout: rgw sync log trim interval = 0 2020-11-19T02:50:48.802 INFO:teuthology.orchestra.run.smithi086.stdout: rgw sync obj etag verify = True 2020-11-19T02:50:48.802 INFO:teuthology.orchestra.run.smithi086.stdout: setgroup = ceph 2020-11-19T02:50:48.802 INFO:teuthology.orchestra.run.smithi086.stdout: setuser = ceph 2020-11-19T02:50:48.803 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2020-11-19T02:50:48.803 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2020-11-19T02:50:48.803 INFO:teuthology.orchestra.run.smithi086.stdout: osd_max_omap_entries_per_request = 10 2020-11-19T02:50:48.803 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2020-11-19T02:50:48.803 INFO:teuthology.orchestra.run.smithi086.stdout: osd_max_omap_entries_per_request = 1000 2020-11-19T02:50:48.804 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2020-11-19T02:50:48.804 INFO:teuthology.orchestra.run.smithi086.stdout: osd_max_omap_entries_per_request = 10000 2020-11-19T02:50:48.805 INFO:teuthology.orchestra.run.smithi160.stdout:[global] 2020-11-19T02:50:48.806 INFO:teuthology.orchestra.run.smithi160.stdout: chdir = "" 2020-11-19T02:50:48.806 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-19T02:50:48.806 INFO:teuthology.orchestra.run.smithi160.stdout: auth supported = cephx 2020-11-19T02:50:48.807 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.807 INFO:teuthology.orchestra.run.smithi160.stdout: filestore xattr use omap = true 2020-11-19T02:50:48.807 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.807 INFO:teuthology.orchestra.run.smithi160.stdout: mon clock drift allowed = 1.000 2020-11-19T02:50:48.807 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.808 INFO:teuthology.orchestra.run.smithi160.stdout: osd crush chooseleaf type = 0 2020-11-19T02:50:48.808 INFO:teuthology.orchestra.run.smithi160.stdout: auth debug = true 2020-11-19T02:50:48.808 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.808 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on old message = true 2020-11-19T02:50:48.808 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on bug = true 2020-11-19T02:50:48.809 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.809 INFO:teuthology.orchestra.run.smithi160.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-19T02:50:48.809 INFO:teuthology.orchestra.run.smithi160.stdout: mon pg warn max object skew = 0 2020-11-19T02:50:48.809 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.810 INFO:teuthology.orchestra.run.smithi160.stdout: # disable pg_autoscaler by default for new pools 2020-11-19T02:50:48.810 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-19T02:50:48.810 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.810 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default size = 2 2020-11-19T02:50:48.811 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.811 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow primary affinity = true 2020-11-19T02:50:48.811 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow pg remap = true 2020-11-19T02:50:48.811 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on legacy crush tunables = false 2020-11-19T02:50:48.811 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on crush straw calc version zero = false 2020-11-19T02:50:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on no sortbitwise = false 2020-11-19T02:50:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on osd down out interval zero = false 2020-11-19T02:50:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on too few osds = false 2020-11-19T02:50:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-19T02:50:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-19T02:50:48.813 INFO:teuthology.orchestra.run.smithi160.stdout: mon_allow_pool_size_one = true 2020-11-19T02:50:48.813 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.813 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-19T02:50:48.813 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.814 INFO:teuthology.orchestra.run.smithi160.stdout: osd default data pool replay window = 5 2020-11-19T02:50:48.814 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.814 INFO:teuthology.orchestra.run.smithi160.stdout: mon allow pool delete = true 2020-11-19T02:50:48.814 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.815 INFO:teuthology.orchestra.run.smithi160.stdout: mon cluster log file level = debug 2020-11-19T02:50:48.815 INFO:teuthology.orchestra.run.smithi160.stdout: debug asserts on shutdown = true 2020-11-19T02:50:48.815 INFO:teuthology.orchestra.run.smithi160.stdout: mon health detail to clog = false 2020-11-19T02:50:48.815 INFO:teuthology.orchestra.run.smithi160.stdout: mon host = 172.21.15.86 2020-11-19T02:50:48.815 INFO:teuthology.orchestra.run.smithi160.stdout: osd heartbeat grace = 40 2020-11-19T02:50:48.816 INFO:teuthology.orchestra.run.smithi160.stdout: fsid = b27f53c1-7dd2-4d8a-83e5-41e60c239cf7 2020-11-19T02:50:48.816 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.816 INFO:teuthology.orchestra.run.smithi160.stdout:[osd] 2020-11-19T02:50:48.816 INFO:teuthology.orchestra.run.smithi160.stdout: osd journal size = 100 2020-11-19T02:50:48.816 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.817 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub load threshold = 5.0 2020-11-19T02:50:48.817 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub max interval = 600 2020-11-19T02:50:48.817 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.817 INFO:teuthology.orchestra.run.smithi160.stdout: osd recover clone overlap = true 2020-11-19T02:50:48.818 INFO:teuthology.orchestra.run.smithi160.stdout: osd recovery max chunk = 1048576 2020-11-19T02:50:48.818 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.818 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug shutdown = true 2020-11-19T02:50:48.818 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug op order = true 2020-11-19T02:50:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify stray on activate = true 2020-11-19T02:50:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: osd open classes on start = true 2020-11-19T02:50:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug pg log writeout = true 2020-11-19T02:50:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.820 INFO:teuthology.orchestra.run.smithi160.stdout: osd deep scrub update digest min age = 30 2020-11-19T02:50:48.820 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.820 INFO:teuthology.orchestra.run.smithi160.stdout: osd map max advance = 10 2020-11-19T02:50:48.820 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.820 INFO:teuthology.orchestra.run.smithi160.stdout: journal zero on create = true 2020-11-19T02:50:48.821 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.821 INFO:teuthology.orchestra.run.smithi160.stdout: filestore ondisk finisher threads = 3 2020-11-19T02:50:48.821 INFO:teuthology.orchestra.run.smithi160.stdout: filestore apply finisher threads = 3 2020-11-19T02:50:48.822 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.822 INFO:teuthology.orchestra.run.smithi160.stdout: bdev debug aio = true 2020-11-19T02:50:48.822 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug misdirected ops = true 2020-11-19T02:50:48.822 INFO:teuthology.orchestra.run.smithi160.stdout: debug filestore = 20 2020-11-19T02:50:48.822 INFO:teuthology.orchestra.run.smithi160.stdout: debug journal = 20 2020-11-19T02:50:48.823 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 20 2020-11-19T02:50:48.823 INFO:teuthology.orchestra.run.smithi160.stdout: debug osd = 25 2020-11-19T02:50:48.823 INFO:teuthology.orchestra.run.smithi160.stdout: osd fast shutdown = False 2020-11-19T02:50:48.823 INFO:teuthology.orchestra.run.smithi160.stdout: rocksdb delete range threshold = 0 2020-11-19T02:50:48.823 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.824 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr] 2020-11-19T02:50:48.824 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2020-11-19T02:50:48.824 INFO:teuthology.orchestra.run.smithi160.stdout: debug mgr = 20 2020-11-19T02:50:48.824 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2020-11-19T02:50:48.825 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2020-11-19T02:50:48.825 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2020-11-19T02:50:48.825 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2020-11-19T02:50:48.825 INFO:teuthology.orchestra.run.smithi160.stdout: mgr/telemetry/nag = false 2020-11-19T02:50:48.826 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.826 INFO:teuthology.orchestra.run.smithi160.stdout:[mon] 2020-11-19T02:50:48.826 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2020-11-19T02:50:48.826 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2020-11-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi160.stdout: debug paxos = 20 2020-11-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2020-11-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi160.stdout: mon data avail warn = 5 2020-11-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi160.stdout: mon mgr mkfs grace = 240 2020-11-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2020-11-19T02:50:48.828 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd reporter subtree level = osd 2020-11-19T02:50:48.828 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd prime pg temp = true 2020-11-19T02:50:48.828 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2020-11-19T02:50:48.828 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd crush smoke test = False 2020-11-19T02:50:48.829 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:50:48.829 INFO:teuthology.orchestra.run.smithi160.stdout:[client] 2020-11-19T02:50:48.829 INFO:teuthology.orchestra.run.smithi160.stdout: rgw cache enabled = true 2020-11-19T02:50:48.829 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable ops log = true 2020-11-19T02:50:48.830 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable usage log = true 2020-11-19T02:50:48.830 INFO:teuthology.orchestra.run.smithi160.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-19T02:50:48.830 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-19T02:50:48.830 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw = 20 2020-11-19T02:50:48.830 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt require ssl = False 2020-11-19T02:50:48.831 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms backend = testing 2020-11-19T02:50:48.831 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T02:50:48.831 INFO:teuthology.orchestra.run.smithi160.stdout: rgw curl low speed time = 300 2020-11-19T02:50:48.831 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data log num shards = 4 2020-11-19T02:50:48.831 INFO:teuthology.orchestra.run.smithi160.stdout: rgw md log max shards = 4 2020-11-19T02:50:48.832 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync log trim interval = 0 2020-11-19T02:50:48.832 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync obj etag verify = True 2020-11-19T02:50:48.832 INFO:teuthology.orchestra.run.smithi160.stdout: setgroup = ceph 2020-11-19T02:50:48.832 INFO:teuthology.orchestra.run.smithi160.stdout: setuser = ceph 2020-11-19T02:50:48.833 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.a] 2020-11-19T02:50:48.833 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2020-11-19T02:50:48.833 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10 2020-11-19T02:50:48.833 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2020-11-19T02:50:48.834 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 1000 2020-11-19T02:50:48.834 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2020-11-19T02:50:48.834 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10000 2020-11-19T02:50:48.834 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-11-19T02:50:48.835 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-11-19T02:50:48.893 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-19T02:50:48.894 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:48.894 INFO:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-19T02:50:48.898 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-19T02:50:48.899 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.] 2020-11-19T02:50:48.899 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCI3bVfZsxFGhAAeL6VZKJwPhjKtTUoUMjX7w== 2020-11-19T02:50:48.899 INFO:teuthology.orchestra.run.smithi086.stdout:[client.admin] 2020-11-19T02:50:48.900 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCI3bVfX3IbNRAAeRMzCYtdSjmPROYByrFHhg== 2020-11-19T02:50:48.900 INFO:teuthology.orchestra.run.smithi086.stdout: caps mds = "allow *" 2020-11-19T02:50:48.900 INFO:teuthology.orchestra.run.smithi086.stdout: caps mgr = "allow *" 2020-11-19T02:50:48.900 INFO:teuthology.orchestra.run.smithi086.stdout: caps mon = "allow *" 2020-11-19T02:50:48.901 INFO:teuthology.orchestra.run.smithi086.stdout: caps osd = "allow *" 2020-11-19T02:50:48.901 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:48.901 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:48.901 INFO:teuthology.orchestra.run.smithi086.stderr:206 bytes copied, 7.8691e-05 s, 2.6 MB/s 2020-11-19T02:50:48.902 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:48.902 INFO:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-19T02:50:48.944 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-19T02:50:48.946 INFO:teuthology.orchestra.run.smithi086.stdout: ��S�}�M���A� #���ݵ_�O&�ݵ_�O&�aXa �V��Va 2020-11-19T02:50:48.946 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:48.947 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:48.947 INFO:teuthology.orchestra.run.smithi086.stderr:201 bytes copied, 0.000159061 s, 1.3 MB/s 2020-11-19T02:50:48.947 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2020-11-19T02:50:48.948 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:48.948 INFO:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/c1.keyring 2020-11-19T02:50:48.948 INFO:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T02:50:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-19T02:50:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:206 bytes copied, 0.000121634 s, 1.7 MB/s 2020-11-19T02:50:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T02:50:49.012 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:49.013 INFO:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:49.061 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:49.062 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:49.063 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:49.063 INFO:teuthology.orchestra.run.smithi086.stderr:201 bytes copied, 0.000128127 s, 1.6 MB/s 2020-11-19T02:50:49.063 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi160.front.sepia.ceph.com 2020-11-19T02:50:49.063 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:50:49.064 INFO:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/c1.keyring 2020-11-19T02:50:49.064 INFO:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T02:50:49.069 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-19T02:50:49.078 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:50:49.079 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:50:49.079 INFO:teuthology.orchestra.run.smithi160.stderr:206 bytes copied, 7.4337e-05 s, 2.8 MB/s 2020-11-19T02:50:49.080 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T02:50:49.088 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:50:49.089 INFO:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:49.134 INFO:teuthology.orchestra.run.smithi160.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:49.135 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:50:49.136 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:50:49.136 INFO:teuthology.orchestra.run.smithi160.stderr:201 bytes copied, 7.8244e-05 s, 2.6 MB/s 2020-11-19T02:50:49.136 INFO:tasks.ceph:Setting up mon nodes... 2020-11-19T02:50:49.137 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-19T02:50:49.137 INFO:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-11-19T02:50:49.190 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-11-19T02:50:49.192 INFO:tasks.ceph:Setting up mds nodes... 2020-11-19T02:50:49.193 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-19T02:50:49.194 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-11-19T02:50:49.273 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/c1.client.0.keyring 2020-11-19T02:50:49.288 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-11-19T02:50:49.366 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/c1.client.1.keyring 2020-11-19T02:50:49.378 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-19T02:50:49.378 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-19T02:50:49.379 INFO:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-11-19T02:50:49.435 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T02:50:49.436 INFO:tasks.ceph:role: c1.osd.0 2020-11-19T02:50:49.436 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2020-11-19T02:50:49.437 INFO:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-19T02:50:49.959 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T02:50:49.960 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T02:50:49.960 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T02:50:49.960 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T02:50:49.960 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2020-11-19T02:50:49.960 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T02:50:49.961 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T02:50:49.961 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T02:50:49.961 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T02:50:49.962 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2020-11-19T02:50:49.962 INFO:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-11-19T02:50:50.074 INFO:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-11-19T02:50:50.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T02:50:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T02:50:50.088 INFO:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-11-19T02:50:50.148 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T02:50:50.149 INFO:tasks.ceph:role: c1.osd.1 2020-11-19T02:50:50.149 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2020-11-19T02:50:50.150 INFO:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-19T02:50:50.665 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T02:50:50.665 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T02:50:50.665 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T02:50:50.666 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T02:50:50.666 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2020-11-19T02:50:50.666 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T02:50:50.666 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T02:50:50.666 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T02:50:50.667 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T02:50:50.668 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2020-11-19T02:50:50.668 INFO:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-11-19T02:50:50.721 INFO:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-11-19T02:50:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T02:50:50.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T02:50:50.782 INFO:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-11-19T02:50:50.844 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T02:50:51.170 INFO:tasks.ceph:role: c1.osd.2 2020-11-19T02:50:51.172 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2020-11-19T02:50:51.174 INFO:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-19T02:50:51.661 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T02:50:51.661 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T02:50:51.662 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T02:50:51.663 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T02:50:51.663 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2020-11-19T02:50:51.664 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T02:50:51.665 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T02:50:51.666 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T02:50:51.666 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T02:50:51.669 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2020-11-19T02:50:51.670 INFO:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-11-19T02:50:51.721 INFO:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-11-19T02:50:51.734 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T02:50:51.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T02:50:51.736 INFO:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:51.820+0000 7f2966fb2d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-11-19T02:50:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:51.820+0000 7f2966fb2d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-11-19T02:50:51.829 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:51.820+0000 7f2966fb2d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-19T02:50:51.829 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:51.820+0000 7f2966fb2d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-19T02:50:51.829 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:51.820+0000 7f2966fb2d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-19T02:50:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:51.824+0000 7f2966fb2d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-11-19T02:50:54.009 INFO:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-11-19T02:50:54.025 INFO:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:54.109 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:54.100+0000 7f5874b50d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-11-19T02:50:54.109 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:54.104+0000 7f5874b50d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-11-19T02:50:54.109 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:54.104+0000 7f5874b50d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-19T02:50:54.110 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:54.104+0000 7f5874b50d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-19T02:50:54.110 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:54.104+0000 7f5874b50d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-19T02:50:54.110 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:54.104+0000 7f5874b50d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-11-19T02:50:56.273 INFO:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-11-19T02:50:56.333 INFO:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:56.417 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:56.408+0000 7fd9138e1d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-11-19T02:50:56.417 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:56.408+0000 7fd9138e1d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-11-19T02:50:56.418 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:56.408+0000 7fd9138e1d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-19T02:50:56.418 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:56.408+0000 7fd9138e1d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-19T02:50:56.418 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:56.408+0000 7fd9138e1d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-19T02:50:56.419 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:50:56.412+0000 7fd9138e1d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-11-19T02:50:58.592 INFO:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-11-19T02:50:58.653 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-19T02:50:58.653 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:58.654 INFO:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-19T02:50:58.702 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-19T02:50:58.711 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2020-11-19T02:50:58.711 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCJ3bVf1MdMCxAASTBY6DYJXvDJ8PeSqFBs7Q== 2020-11-19T02:50:58.712 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:58.712 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:58.712 INFO:teuthology.orchestra.run.smithi086.stderr:56 bytes copied, 0.00017702 s, 316 kB/s 2020-11-19T02:50:58.713 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:58.714 INFO:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-19T02:50:58.763 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-19T02:50:58.770 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2020-11-19T02:50:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCL3bVfObtNMRAA99S6Yn+T/9iKYwexPkXsuA== 2020-11-19T02:50:58.771 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:58.771 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:58.772 INFO:teuthology.orchestra.run.smithi086.stderr:56 bytes copied, 0.000160073 s, 350 kB/s 2020-11-19T02:50:58.772 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:58.773 INFO:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-19T02:50:58.822 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-19T02:50:58.831 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2020-11-19T02:50:58.831 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCO3bVfAml0BhAAAFdx4M8LYO2jjHroy5mOqg== 2020-11-19T02:50:58.832 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:58.832 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:58.832 INFO:teuthology.orchestra.run.smithi086.stderr:56 bytes copied, 0.000161427 s, 347 kB/s 2020-11-19T02:50:58.833 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:58.833 INFO:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-19T02:50:58.882 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-19T02:50:58.891 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2020-11-19T02:50:58.892 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCQ3bVfOX7GGBAA/OiJkQrxkijTjfNrbRXQpg== 2020-11-19T02:50:58.892 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:58.893 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:58.893 INFO:teuthology.orchestra.run.smithi086.stderr:56 bytes copied, 0.000168987 s, 331 kB/s 2020-11-19T02:50:58.894 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:58.894 INFO:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-19T02:50:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-19T02:50:58.938 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2020-11-19T02:50:58.938 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCJ3bVfE1k+EBAACeM8K/tp4QSir6bpYBKT3A== 2020-11-19T02:50:58.939 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:58.939 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:59 bytes copied, 0.000118011 s, 500 kB/s 2020-11-19T02:50:58.940 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:50:58.941 INFO:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-19T02:50:58.984 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-19T02:50:58.986 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2020-11-19T02:50:58.986 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCJ3bVfCli6FRAAGSvI3Xbp7yY2W0oQ076n8Q== 2020-11-19T02:50:58.986 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:50:58.987 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:50:58.987 INFO:teuthology.orchestra.run.smithi086.stderr:59 bytes copied, 0.000167806 s, 352 kB/s 2020-11-19T02:50:58.987 INFO:tasks.ceph:Adding keys to all mons... 2020-11-19T02:50:58.987 INFO:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/c1.keyring 2020-11-19T02:50:59.041 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2020-11-19T02:50:59.042 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCJ3bVf1MdMCxAASTBY6DYJXvDJ8PeSqFBs7Q== 2020-11-19T02:50:59.042 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2020-11-19T02:50:59.042 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCL3bVfObtNMRAA99S6Yn+T/9iKYwexPkXsuA== 2020-11-19T02:50:59.042 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2020-11-19T02:50:59.043 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCO3bVfAml0BhAAAFdx4M8LYO2jjHroy5mOqg== 2020-11-19T02:50:59.043 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2020-11-19T02:50:59.043 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCQ3bVfOX7GGBAA/OiJkQrxkijTjfNrbRXQpg== 2020-11-19T02:50:59.043 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2020-11-19T02:50:59.043 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCJ3bVfE1k+EBAACeM8K/tp4QSir6bpYBKT3A== 2020-11-19T02:50:59.044 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2020-11-19T02:50:59.044 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCJ3bVfCli6FRAAGSvI3Xbp7yY2W0oQ076n8Q== 2020-11-19T02:50:59.045 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-19T02:50:59.133 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T02:50:59.218 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T02:50:59.303 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T02:50:59.387 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-19T02:50:59.475 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-19T02:50:59.604 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-19T02:50:59.606 INFO:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-11-19T02:50:59.628 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-11-19T02:50:59.724 INFO:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-11-19T02:50:59.738 INFO:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-11-19T02:50:59.787 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-11-19T02:50:59.788 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-11-19T02:50:59.789 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-11-19T02:50:59.831 INFO:tasks.ceph.c1.mon.a:Started 2020-11-19T02:50:59.833 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-11-19T02:50:59.834 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-11-19T02:50:59.835 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-11-19T02:50:59.842 INFO:tasks.ceph.c1.mgr.x:Started 2020-11-19T02:50:59.843 INFO:tasks.ceph:Setting crush tunables to default 2020-11-19T02:50:59.843 INFO:teuthology.orchestra.run.smithi086:> sudo ceph --cluster c1 osd crush tunables default 2020-11-19T02:51:00.179 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2020-11-19T02:51:00.192 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-11-19T02:51:00.192 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:51:00.193 INFO:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-19T02:51:00.197 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-19T02:51:00.202 INFO:teuthology.orchestra.run.smithi086.stdout:65b8ddd6-4950-47ce-b57d-0b3cbacd41e4 2020-11-19T02:51:00.203 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:51:00.203 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:51:00.203 INFO:teuthology.orchestra.run.smithi086.stderr:37 bytes copied, 0.000158267 s, 234 kB/s 2020-11-19T02:51:00.204 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:51:00.204 INFO:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-19T02:51:00.248 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-19T02:51:00.257 INFO:teuthology.orchestra.run.smithi086.stdout:eff4fc0d-b343-46c6-b405-71918ca5f2c2 2020-11-19T02:51:00.257 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:51:00.257 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:51:00.258 INFO:teuthology.orchestra.run.smithi086.stderr:37 bytes copied, 8.9687e-05 s, 413 kB/s 2020-11-19T02:51:00.258 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:51:00.259 INFO:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-19T02:51:00.307 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-19T02:51:00.316 INFO:teuthology.orchestra.run.smithi086.stdout:301d867b-6e5e-4322-8c4d-16afdbcc355e 2020-11-19T02:51:00.317 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:51:00.318 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:51:00.318 INFO:teuthology.orchestra.run.smithi086.stderr:37 bytes copied, 0.000100543 s, 368 kB/s 2020-11-19T02:51:00.318 INFO:teuthology.orchestra.run.smithi086:> sudo ceph --cluster c1 osd new 65b8ddd6-4950-47ce-b57d-0b3cbacd41e4 0 2020-11-19T02:51:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:0 2020-11-19T02:51:00.609 INFO:teuthology.orchestra.run.smithi086:> sudo ceph --cluster c1 osd new eff4fc0d-b343-46c6-b405-71918ca5f2c2 1 2020-11-19T02:51:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:1 2020-11-19T02:51:01.907 INFO:teuthology.orchestra.run.smithi086:> sudo ceph --cluster c1 osd new 301d867b-6e5e-4322-8c4d-16afdbcc355e 2 2020-11-19T02:51:02.120 INFO:teuthology.orchestra.run.smithi086.stdout:2 2020-11-19T02:51:02.134 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-11-19T02:51:02.135 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-11-19T02:51:02.137 INFO:tasks.ceph.c1.osd.0:Started 2020-11-19T02:51:02.138 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-11-19T02:51:02.138 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-11-19T02:51:02.140 INFO:tasks.ceph.c1.osd.1:Started 2020-11-19T02:51:02.141 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-11-19T02:51:02.141 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-11-19T02:51:02.145 INFO:tasks.ceph.c1.osd.2:Started 2020-11-19T02:51:02.145 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-19T02:51:02.449 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-19T02:51:02.545 INFO:tasks.ceph.c1.osd.1.smithi086.stderr:2020-11-19T02:51:02.540+0000 7fccd035fd80 -1 Falling back to public interface 2020-11-19T02:51:02.631 INFO:teuthology.misc.health.smithi086.stdout: 2020-11-19T02:51:02.631 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":5,"fsid":"b27f53c1-7dd2-4d8a-83e5-41e60c239cf7","created":"2020-11-19T02:50:59.958524+0000","modified":"2020-11-19T02:51:02.116198+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"65b8ddd6-4950-47ce-b57d-0b3cbacd41e4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff4fc0d-b343-46c6-b405-71918ca5f2c2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"301d867b-6e5e-4322-8c4d-16afdbcc355e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:02.642 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-19T02:51:02.782 INFO:tasks.ceph.c1.osd.2.smithi086.stderr:2020-11-19T02:51:02.776+0000 7f2c89db9d80 -1 Falling back to public interface 2020-11-19T02:51:02.787 INFO:tasks.ceph.c1.osd.0.smithi086.stderr:2020-11-19T02:51:02.780+0000 7f6c4954dd80 -1 Falling back to public interface 2020-11-19T02:51:03.178 INFO:tasks.ceph.c1.osd.2.smithi086.stderr:2020-11-19T02:51:03.172+0000 7f2c89db9d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-19T02:51:03.182 INFO:tasks.ceph.c1.osd.1.smithi086.stderr:2020-11-19T02:51:03.176+0000 7fccd035fd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-19T02:51:03.421 INFO:tasks.ceph.c1.osd.0.smithi086.stderr:2020-11-19T02:51:03.416+0000 7f6c4954dd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-19T02:51:05.193 INFO:tasks.ceph.c1.osd.2.smithi086.stderr:2020-11-19T02:51:05.188+0000 7f2c81283700 -1 osd.2 0 waiting for initial osdmap 2020-11-19T02:51:05.193 INFO:tasks.ceph.c1.osd.0.smithi086.stderr:2020-11-19T02:51:05.188+0000 7f6c4221a700 -1 osd.0 0 waiting for initial osdmap 2020-11-19T02:51:05.194 INFO:tasks.ceph.c1.osd.1.smithi086.stderr:2020-11-19T02:51:05.188+0000 7fccc902c700 -1 osd.1 0 waiting for initial osdmap 2020-11-19T02:51:08.946 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-19T02:51:09.238 INFO:teuthology.misc.health.smithi086.stdout: 2020-11-19T02:51:09.238 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"b27f53c1-7dd2-4d8a-83e5-41e60c239cf7","created":"2020-11-19T02:50:59.958524+0000","modified":"2020-11-19T02:51:08.203429+0000","last_up_change":"2020-11-19T02:51:06.186002+0000","last_in_change":"2020-11-19T02:51:06.186002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T02:51:03.953030+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"65b8ddd6-4950-47ce-b57d-0b3cbacd41e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":11064},{"type":"v1","addr":"172.21.15.86:6817","nonce":11064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":11064},{"type":"v1","addr":"172.21.15.86:6819","nonce":11064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":11064},{"type":"v1","addr":"172.21.15.86:6823","nonce":11064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":11064},{"type":"v1","addr":"172.21.15.86:6821","nonce":11064}]},"public_addr":"172.21.15.86:6817/11064","cluster_addr":"172.21.15.86:6819/11064","heartbeat_back_addr":"172.21.15.86:6823/11064","heartbeat_front_addr":"172.21.15.86:6821/11064","state":["exists","up"]},{"osd":1,"uuid":"eff4fc0d-b343-46c6-b405-71918ca5f2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":11066},{"type":"v1","addr":"172.21.15.86:6801","nonce":11066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":11066},{"type":"v1","addr":"172.21.15.86:6803","nonce":11066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":11066},{"type":"v1","addr":"172.21.15.86:6807","nonce":11066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":11066},{"type":"v1","addr":"172.21.15.86:6805","nonce":11066}]},"public_addr":"172.21.15.86:6801/11066","cluster_addr":"172.21.15.86:6803/11066","heartbeat_back_addr":"172.21.15.86:6807/11066","heartbeat_front_addr":"172.21.15.86:6805/11066","state":["exists","up"]},{"osd":2,"uuid":"301d867b-6e5e-4322-8c4d-16afdbcc355e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":11065},{"type":"v1","addr":"172.21.15.86:6809","nonce":11065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":11065},{"type":"v1","addr":"172.21.15.86:6811","nonce":11065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":11065},{"type":"v1","addr":"172.21.15.86:6815","nonce":11065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":11065},{"type":"v1","addr":"172.21.15.86:6813","nonce":11065}]},"public_addr":"172.21.15.86:6809/11065","cluster_addr":"172.21.15.86:6811/11065","heartbeat_back_addr":"172.21.15.86:6815/11065","heartbeat_front_addr":"172.21.15.86:6813/11065","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.459996+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.159795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.152162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:09.251 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-19T02:51:09.252 INFO:tasks.ceph:Creating RBD pool 2020-11-19T02:51:09.252 INFO:teuthology.orchestra.run.smithi086:> sudo ceph --cluster c1 osd pool create rbd 8 2020-11-19T02:51:10.233 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'rbd' created 2020-11-19T02:51:10.245 INFO:teuthology.orchestra.run.smithi086:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-19T02:51:11.243 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-19T02:51:11.255 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-11-19T02:51:11.255 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-19T02:51:11.256 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-19T02:51:11.539 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:11.539 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":12,"fsid":"b27f53c1-7dd2-4d8a-83e5-41e60c239cf7","created":"2020-11-19T02:50:59.958524+0000","modified":"2020-11-19T02:51:11.220412+0000","last_up_change":"2020-11-19T02:51:06.186002+0000","last_in_change":"2020-11-19T02:51:06.186002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T02:51:03.953030+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T02:51:09.536290+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"65b8ddd6-4950-47ce-b57d-0b3cbacd41e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":11064},{"type":"v1","addr":"172.21.15.86:6817","nonce":11064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":11064},{"type":"v1","addr":"172.21.15.86:6819","nonce":11064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":11064},{"type":"v1","addr":"172.21.15.86:6823","nonce":11064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":11064},{"type":"v1","addr":"172.21.15.86:6821","nonce":11064}]},"public_addr":"172.21.15.86:6817/11064","cluster_addr":"172.21.15.86:6819/11064","heartbeat_back_addr":"172.21.15.86:6823/11064","heartbeat_front_addr":"172.21.15.86:6821/11064","state":["exists","up"]},{"osd":1,"uuid":"eff4fc0d-b343-46c6-b405-71918ca5f2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":11066},{"type":"v1","addr":"172.21.15.86:6801","nonce":11066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":11066},{"type":"v1","addr":"172.21.15.86:6803","nonce":11066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":11066},{"type":"v1","addr":"172.21.15.86:6807","nonce":11066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":11066},{"type":"v1","addr":"172.21.15.86:6805","nonce":11066}]},"public_addr":"172.21.15.86:6801/11066","cluster_addr":"172.21.15.86:6803/11066","heartbeat_back_addr":"172.21.15.86:6807/11066","heartbeat_front_addr":"172.21.15.86:6805/11066","state":["exists","up"]},{"osd":2,"uuid":"301d867b-6e5e-4322-8c4d-16afdbcc355e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":11065},{"type":"v1","addr":"172.21.15.86:6809","nonce":11065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":11065},{"type":"v1","addr":"172.21.15.86:6811","nonce":11065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":11065},{"type":"v1","addr":"172.21.15.86:6815","nonce":11065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":11065},{"type":"v1","addr":"172.21.15.86:6813","nonce":11065}]},"public_addr":"172.21.15.86:6809/11065","cluster_addr":"172.21.15.86:6811/11065","heartbeat_back_addr":"172.21.15.86:6815/11065","heartbeat_front_addr":"172.21.15.86:6813/11065","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.459996+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.159795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.152162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:11.558 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-19T02:51:03.953030+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-19T02:51:09.536290+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-19T02:51:11.559 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2020-11-19T02:51:11.856 INFO:teuthology.orchestra.run.smithi086.stdout:pg_num: 1 2020-11-19T02:51:11.869 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2020-11-19T02:51:12.168 INFO:teuthology.orchestra.run.smithi086.stdout:pg_num: 8 2020-11-19T02:51:12.182 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-11-19T02:51:12.183 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-11-19T02:51:12.183 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2020-11-19T02:51:12.512 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:12.533 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":10877},{"type":"v1","addr":"172.21.15.86:6825","nonce":10877}]},"active_addr":"172.21.15.86:6825/10877","active_change":"2020-11-19T02:51:03.926071+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":220087066}]},{"addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":4294278971}]},{"addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2034607700}]}]}} 2020-11-19T02:51:12.536 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-11-19T02:51:12.536 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-11-19T02:51:12.536 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-19T02:51:12.829 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:12.829 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":13,"fsid":"b27f53c1-7dd2-4d8a-83e5-41e60c239cf7","created":"2020-11-19T02:50:59.958524+0000","modified":"2020-11-19T02:51:12.226944+0000","last_up_change":"2020-11-19T02:51:06.186002+0000","last_in_change":"2020-11-19T02:51:06.186002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T02:51:03.953030+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T02:51:09.536290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"65b8ddd6-4950-47ce-b57d-0b3cbacd41e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":11064},{"type":"v1","addr":"172.21.15.86:6817","nonce":11064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":11064},{"type":"v1","addr":"172.21.15.86:6819","nonce":11064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":11064},{"type":"v1","addr":"172.21.15.86:6823","nonce":11064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":11064},{"type":"v1","addr":"172.21.15.86:6821","nonce":11064}]},"public_addr":"172.21.15.86:6817/11064","cluster_addr":"172.21.15.86:6819/11064","heartbeat_back_addr":"172.21.15.86:6823/11064","heartbeat_front_addr":"172.21.15.86:6821/11064","state":["exists","up"]},{"osd":1,"uuid":"eff4fc0d-b343-46c6-b405-71918ca5f2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":11066},{"type":"v1","addr":"172.21.15.86:6801","nonce":11066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":11066},{"type":"v1","addr":"172.21.15.86:6803","nonce":11066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":11066},{"type":"v1","addr":"172.21.15.86:6807","nonce":11066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":11066},{"type":"v1","addr":"172.21.15.86:6805","nonce":11066}]},"public_addr":"172.21.15.86:6801/11066","cluster_addr":"172.21.15.86:6803/11066","heartbeat_back_addr":"172.21.15.86:6807/11066","heartbeat_front_addr":"172.21.15.86:6805/11066","state":["exists","up"]},{"osd":2,"uuid":"301d867b-6e5e-4322-8c4d-16afdbcc355e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":11065},{"type":"v1","addr":"172.21.15.86:6809","nonce":11065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":11065},{"type":"v1","addr":"172.21.15.86:6811","nonce":11065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":11065},{"type":"v1","addr":"172.21.15.86:6815","nonce":11065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":11065},{"type":"v1","addr":"172.21.15.86:6813","nonce":11065}]},"public_addr":"172.21.15.86:6809/11065","cluster_addr":"172.21.15.86:6811/11065","heartbeat_back_addr":"172.21.15.86:6815/11065","heartbeat_front_addr":"172.21.15.86:6813/11065","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.459996+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.159795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.152162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:12.843 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-11-19T02:51:12.843 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-19T02:51:13.137 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:13.137 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":13,"fsid":"b27f53c1-7dd2-4d8a-83e5-41e60c239cf7","created":"2020-11-19T02:50:59.958524+0000","modified":"2020-11-19T02:51:12.226944+0000","last_up_change":"2020-11-19T02:51:06.186002+0000","last_in_change":"2020-11-19T02:51:06.186002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T02:51:03.953030+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T02:51:09.536290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"65b8ddd6-4950-47ce-b57d-0b3cbacd41e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":11064},{"type":"v1","addr":"172.21.15.86:6817","nonce":11064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":11064},{"type":"v1","addr":"172.21.15.86:6819","nonce":11064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":11064},{"type":"v1","addr":"172.21.15.86:6823","nonce":11064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":11064},{"type":"v1","addr":"172.21.15.86:6821","nonce":11064}]},"public_addr":"172.21.15.86:6817/11064","cluster_addr":"172.21.15.86:6819/11064","heartbeat_back_addr":"172.21.15.86:6823/11064","heartbeat_front_addr":"172.21.15.86:6821/11064","state":["exists","up"]},{"osd":1,"uuid":"eff4fc0d-b343-46c6-b405-71918ca5f2c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":11066},{"type":"v1","addr":"172.21.15.86:6801","nonce":11066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":11066},{"type":"v1","addr":"172.21.15.86:6803","nonce":11066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":11066},{"type":"v1","addr":"172.21.15.86:6807","nonce":11066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":11066},{"type":"v1","addr":"172.21.15.86:6805","nonce":11066}]},"public_addr":"172.21.15.86:6801/11066","cluster_addr":"172.21.15.86:6803/11066","heartbeat_back_addr":"172.21.15.86:6807/11066","heartbeat_front_addr":"172.21.15.86:6805/11066","state":["exists","up"]},{"osd":2,"uuid":"301d867b-6e5e-4322-8c4d-16afdbcc355e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":11065},{"type":"v1","addr":"172.21.15.86:6809","nonce":11065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":11065},{"type":"v1","addr":"172.21.15.86:6811","nonce":11065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":11065},{"type":"v1","addr":"172.21.15.86:6815","nonce":11065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":11065},{"type":"v1","addr":"172.21.15.86:6813","nonce":11065}]},"public_addr":"172.21.15.86:6809/11065","cluster_addr":"172.21.15.86:6811/11065","heartbeat_back_addr":"172.21.15.86:6815/11065","heartbeat_front_addr":"172.21.15.86:6813/11065","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.459996+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.159795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:04.152162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:13.152 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2020-11-19T02:51:13.319 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2020-11-19T02:51:13.320 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2020-11-19T02:51:13.489 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2020-11-19T02:51:13.876 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2020-11-19T02:51:14.053 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2020-11-19T02:51:14.054 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-19T02:51:14.349 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2020-11-19T02:51:14.363 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2020-11-19T02:51:14.364 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2020-11-19T02:51:14.656 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2020-11-19T02:51:14.675 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2020-11-19T02:51:14.676 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2020-11-19T02:51:14.972 INFO:teuthology.orchestra.run.smithi086.stdout:34359738370 2020-11-19T02:51:14.986 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.2 2020-11-19T02:51:15.987 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2020-11-19T02:51:16.291 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2020-11-19T02:51:16.305 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.2 2020-11-19T02:51:16.306 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-11-19T02:51:16.307 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-19T02:51:16.599 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:16.600 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2020-11-19T02:51:16.611 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-11-19T02:51:15.936639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":97095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000946"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.240448+0000","last_change":"2020-11-19T02:51:11.238261+0000","last_active":"2020-11-19T02:51:11.240448+0000","last_peered":"2020-11-19T02:51:11.240448+0000","last_clean":"2020-11-19T02:51:11.240448+0000","last_became_active":"2020-11-19T02:51:11.237802+0000","last_became_peered":"2020-11-19T02:51:11.237802+0000","last_unstale":"2020-11-19T02:51:11.240448+0000","last_undegraded":"2020-11-19T02:51:11.240448+0000","last_fullsized":"2020-11-19T02:51:11.240448+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.241207+0000","last_change":"2020-11-19T02:51:11.240390+0000","last_active":"2020-11-19T02:51:11.241207+0000","last_peered":"2020-11-19T02:51:11.241207+0000","last_clean":"2020-11-19T02:51:11.241207+0000","last_became_active":"2020-11-19T02:51:11.239993+0000","last_became_peered":"2020-11-19T02:51:11.239993+0000","last_unstale":"2020-11-19T02:51:11.241207+0000","last_undegraded":"2020-11-19T02:51:11.241207+0000","last_fullsized":"2020-11-19T02:51:11.241207+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.240494+0000","last_change":"2020-11-19T02:51:11.237999+0000","last_active":"2020-11-19T02:51:11.240494+0000","last_peered":"2020-11-19T02:51:11.240494+0000","last_clean":"2020-11-19T02:51:11.240494+0000","last_became_active":"2020-11-19T02:51:11.237398+0000","last_became_peered":"2020-11-19T02:51:11.237398+0000","last_unstale":"2020-11-19T02:51:11.240494+0000","last_undegraded":"2020-11-19T02:51:11.240494+0000","last_fullsized":"2020-11-19T02:51:11.240494+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.240484+0000","last_change":"2020-11-19T02:51:11.239618+0000","last_active":"2020-11-19T02:51:11.240484+0000","last_peered":"2020-11-19T02:51:11.240484+0000","last_clean":"2020-11-19T02:51:11.240484+0000","last_became_active":"2020-11-19T02:51:11.239182+0000","last_became_peered":"2020-11-19T02:51:11.239182+0000","last_unstale":"2020-11-19T02:51:11.240484+0000","last_undegraded":"2020-11-19T02:51:11.240484+0000","last_fullsized":"2020-11-19T02:51:11.240484+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.234823+0000","last_change":"2020-11-19T02:51:07.408161+0000","last_active":"2020-11-19T02:51:11.234823+0000","last_peered":"2020-11-19T02:51:11.234823+0000","last_clean":"2020-11-19T02:51:11.234823+0000","last_became_active":"2020-11-19T02:51:07.406670+0000","last_became_peered":"2020-11-19T02:51:07.406670+0000","last_unstale":"2020-11-19T02:51:11.234823+0000","last_undegraded":"2020-11-19T02:51:11.234823+0000","last_fullsized":"2020-11-19T02:51:11.234823+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":6,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:04.168990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:04.168990+0000","last_clean_scrub_stamp":"2020-11-19T02:51:04.168990+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.243717+0000","last_change":"2020-11-19T02:51:11.241649+0000","last_active":"2020-11-19T02:51:11.243717+0000","last_peered":"2020-11-19T02:51:11.243717+0000","last_clean":"2020-11-19T02:51:11.243717+0000","last_became_active":"2020-11-19T02:51:11.241204+0000","last_became_peered":"2020-11-19T02:51:11.241204+0000","last_unstale":"2020-11-19T02:51:11.243717+0000","last_undegraded":"2020-11-19T02:51:11.243717+0000","last_fullsized":"2020-11-19T02:51:11.243717+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.239784+0000","last_change":"2020-11-19T02:51:11.236710+0000","last_active":"2020-11-19T02:51:11.239784+0000","last_peered":"2020-11-19T02:51:11.239784+0000","last_clean":"2020-11-19T02:51:11.239784+0000","last_became_active":"2020-11-19T02:51:11.236123+0000","last_became_peered":"2020-11-19T02:51:11.236123+0000","last_unstale":"2020-11-19T02:51:11.239784+0000","last_undegraded":"2020-11-19T02:51:11.239784+0000","last_fullsized":"2020-11-19T02:51:11.239784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.243654+0000","last_change":"2020-11-19T02:51:11.242243+0000","last_active":"2020-11-19T02:51:11.243654+0000","last_peered":"2020-11-19T02:51:11.243654+0000","last_clean":"2020-11-19T02:51:11.243654+0000","last_became_active":"2020-11-19T02:51:11.241606+0000","last_became_peered":"2020-11-19T02:51:11.241606+0000","last_unstale":"2020-11-19T02:51:11.243654+0000","last_undegraded":"2020-11-19T02:51:11.243654+0000","last_fullsized":"2020-11-19T02:51:11.243654+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.240862+0000","last_change":"2020-11-19T02:51:11.239163+0000","last_active":"2020-11-19T02:51:11.240862+0000","last_peered":"2020-11-19T02:51:11.240862+0000","last_clean":"2020-11-19T02:51:11.240862+0000","last_became_active":"2020-11-19T02:51:11.238761+0000","last_became_peered":"2020-11-19T02:51:11.238761+0000","last_unstale":"2020-11-19T02:51:11.240862+0000","last_undegraded":"2020-11-19T02:51:11.240862+0000","last_fullsized":"2020-11-19T02:51:11.240862+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":30023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]}]}]}} 2020-11-19T02:51:16.612 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-19T02:51:16.895 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2020-11-19T02:51:16.906 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-11-19T02:51:15.936639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":97095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000946"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.240448+0000","last_change":"2020-11-19T02:51:11.238261+0000","last_active":"2020-11-19T02:51:11.240448+0000","last_peered":"2020-11-19T02:51:11.240448+0000","last_clean":"2020-11-19T02:51:11.240448+0000","last_became_active":"2020-11-19T02:51:11.237802+0000","last_became_peered":"2020-11-19T02:51:11.237802+0000","last_unstale":"2020-11-19T02:51:11.240448+0000","last_undegraded":"2020-11-19T02:51:11.240448+0000","last_fullsized":"2020-11-19T02:51:11.240448+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.241207+0000","last_change":"2020-11-19T02:51:11.240390+0000","last_active":"2020-11-19T02:51:11.241207+0000","last_peered":"2020-11-19T02:51:11.241207+0000","last_clean":"2020-11-19T02:51:11.241207+0000","last_became_active":"2020-11-19T02:51:11.239993+0000","last_became_peered":"2020-11-19T02:51:11.239993+0000","last_unstale":"2020-11-19T02:51:11.241207+0000","last_undegraded":"2020-11-19T02:51:11.241207+0000","last_fullsized":"2020-11-19T02:51:11.241207+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.240494+0000","last_change":"2020-11-19T02:51:11.237999+0000","last_active":"2020-11-19T02:51:11.240494+0000","last_peered":"2020-11-19T02:51:11.240494+0000","last_clean":"2020-11-19T02:51:11.240494+0000","last_became_active":"2020-11-19T02:51:11.237398+0000","last_became_peered":"2020-11-19T02:51:11.237398+0000","last_unstale":"2020-11-19T02:51:11.240494+0000","last_undegraded":"2020-11-19T02:51:11.240494+0000","last_fullsized":"2020-11-19T02:51:11.240494+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.240484+0000","last_change":"2020-11-19T02:51:11.239618+0000","last_active":"2020-11-19T02:51:11.240484+0000","last_peered":"2020-11-19T02:51:11.240484+0000","last_clean":"2020-11-19T02:51:11.240484+0000","last_became_active":"2020-11-19T02:51:11.239182+0000","last_became_peered":"2020-11-19T02:51:11.239182+0000","last_unstale":"2020-11-19T02:51:11.240484+0000","last_undegraded":"2020-11-19T02:51:11.240484+0000","last_fullsized":"2020-11-19T02:51:11.240484+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.234823+0000","last_change":"2020-11-19T02:51:07.408161+0000","last_active":"2020-11-19T02:51:11.234823+0000","last_peered":"2020-11-19T02:51:11.234823+0000","last_clean":"2020-11-19T02:51:11.234823+0000","last_became_active":"2020-11-19T02:51:07.406670+0000","last_became_peered":"2020-11-19T02:51:07.406670+0000","last_unstale":"2020-11-19T02:51:11.234823+0000","last_undegraded":"2020-11-19T02:51:11.234823+0000","last_fullsized":"2020-11-19T02:51:11.234823+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":6,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:04.168990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:04.168990+0000","last_clean_scrub_stamp":"2020-11-19T02:51:04.168990+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.243717+0000","last_change":"2020-11-19T02:51:11.241649+0000","last_active":"2020-11-19T02:51:11.243717+0000","last_peered":"2020-11-19T02:51:11.243717+0000","last_clean":"2020-11-19T02:51:11.243717+0000","last_became_active":"2020-11-19T02:51:11.241204+0000","last_became_peered":"2020-11-19T02:51:11.241204+0000","last_unstale":"2020-11-19T02:51:11.243717+0000","last_undegraded":"2020-11-19T02:51:11.243717+0000","last_fullsized":"2020-11-19T02:51:11.243717+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.239784+0000","last_change":"2020-11-19T02:51:11.236710+0000","last_active":"2020-11-19T02:51:11.239784+0000","last_peered":"2020-11-19T02:51:11.239784+0000","last_clean":"2020-11-19T02:51:11.239784+0000","last_became_active":"2020-11-19T02:51:11.236123+0000","last_became_peered":"2020-11-19T02:51:11.236123+0000","last_unstale":"2020-11-19T02:51:11.239784+0000","last_undegraded":"2020-11-19T02:51:11.239784+0000","last_fullsized":"2020-11-19T02:51:11.239784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.243654+0000","last_change":"2020-11-19T02:51:11.242243+0000","last_active":"2020-11-19T02:51:11.243654+0000","last_peered":"2020-11-19T02:51:11.243654+0000","last_clean":"2020-11-19T02:51:11.243654+0000","last_became_active":"2020-11-19T02:51:11.241606+0000","last_became_peered":"2020-11-19T02:51:11.241606+0000","last_unstale":"2020-11-19T02:51:11.243654+0000","last_undegraded":"2020-11-19T02:51:11.243654+0000","last_fullsized":"2020-11-19T02:51:11.243654+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:11.240862+0000","last_change":"2020-11-19T02:51:11.239163+0000","last_active":"2020-11-19T02:51:11.240862+0000","last_peered":"2020-11-19T02:51:11.240862+0000","last_clean":"2020-11-19T02:51:11.240862+0000","last_became_active":"2020-11-19T02:51:11.238761+0000","last_became_peered":"2020-11-19T02:51:11.238761+0000","last_unstale":"2020-11-19T02:51:11.240862+0000","last_undegraded":"2020-11-19T02:51:11.240862+0000","last_fullsized":"2020-11-19T02:51:11.240862+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:10.212366+0000","last_clean_scrub_stamp":"2020-11-19T02:51:10.212366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":30023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]}]}]}} 2020-11-19T02:51:16.907 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-11-19T02:51:16.907 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-11-19T02:51:16.908 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-11-19T02:51:16.908 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2020-11-19T02:51:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:17.272 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-19T02:51:17.272 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-11-19T02:51:17.273 INFO:teuthology.run_tasks:Running task ceph... 2020-11-19T02:51:17.286 INFO:tasks.ceph:Creating ceph cluster c2... 2020-11-19T02:51:17.286 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-19T02:51:17.287 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633829', 'branch': 'wip-yuval-fix-47904', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5633829', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuval-fix-47904'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuval-fix-47904', 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuval-fix-47904', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.6672', 'targets': {'smithi086.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOX2yzewpn2i4e07+3TrL99b0P2l1x7R1S9+FHleR/gamdR1+xV0/Rm+YdgPbTxuw+O/rQODT7fLlVl6UBihJS2pfoS+X9X0o1znO23OYTzkehepKz6xqNXkHKnxEdAvddqlisgyrYnC5j6KNiWIlSybv25VJ//uZQqrkKU74Uz2fp6l83hoQjttdrZoowG1KmXEz5gdqczArtjOO7oL81jf7RAY5Abrs0XdnVq0ifxD6lSpsxlv3HiEO5/E4FhEErv5xcawzDG8Nle9rjLjnO9QNp6H87gLyfTj72zE0fgSHbB5F08vO/AxZhZynig/FWjffG2ftCmO9tOywwHVz9', 'smithi160.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDwLcfm2rpI0BEMcouJXOuPRNnfF7ta3SKliynVLHNZbLhzUxe85D2ZtcpDnU2EwXhCEv37c+wXgbEu92FsekOyP246U5k4aX74T8LiGLHuvCXgyvqwq9qjWsDFS4oYn6yuCUzv9aVMpkX6K8eq7vmknRVFEA3Xf3hWWIUeltVOeapmEutN6AlNm+dRfX8rNqiYubr8cWyr5nHsDx6uhjTJBnu1HDghlbttSZNPzeOy95IRTqQUXJHk0G+zBjG5wf+FaeeReGIj/ttMqUQsyHgT5U+zjyGJm0fCTL7ZQF4Rh0e0x+Bj4cq1FLcvNUSXUHmH8q8o0B40fXWovLgRPNbh'}} 2020-11-19T02:51:17.287 INFO:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-19T02:51:17.290 INFO:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-19T02:51:17.298 INFO:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2020-11-19T02:51:17.338 INFO:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/run/ceph 2020-11-19T02:51:17.356 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:17.357 INFO:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2020-11-19T02:51:17.402 INFO:teuthology.orchestra.run.smithi160.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-19T02:51:17.403 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/vg_nvme/lv_1 2020-11-19T02:51:17.403 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/vg_nvme/lv_2 2020-11-19T02:51:17.403 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/vg_nvme/lv_3 2020-11-19T02:51:17.404 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/vg_nvme/lv_4 2020-11-19T02:51:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:17.404 INFO:teuthology.orchestra.run.smithi160.stderr:72 bytes copied, 0.000172763 s, 417 kB/s 2020-11-19T02:51:17.405 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-19T02:51:17.405 INFO:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-19T02:51:17.451 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-19T02:51:17.452 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T02:51:17.452 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-11-19T02:51:17.452 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T02:51:17.452 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2020-11-19 02:47:57.111324039 +0000 2020-11-19T02:51:17.453 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2020-11-19 02:47:57.007272040 +0000 2020-11-19T02:51:17.453 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2020-11-19 02:47:57.007272040 +0000 2020-11-19T02:51:17.453 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2020-11-19T02:51:17.460 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2020-11-19T02:51:17.460 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2020-11-19T02:51:17.461 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000311735 s, 1.6 MB/s 2020-11-19T02:51:17.464 INFO:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-19T02:51:17.511 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-19T02:51:17.512 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T02:51:17.512 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-11-19T02:51:17.512 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T02:51:17.512 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2020-11-19 02:47:57.375456037 +0000 2020-11-19T02:51:17.513 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2020-11-19 02:47:57.263400038 +0000 2020-11-19T02:51:17.513 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2020-11-19 02:47:57.263400038 +0000 2020-11-19T02:51:17.513 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2020-11-19T02:51:17.521 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2020-11-19T02:51:17.521 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2020-11-19T02:51:17.522 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000352287 s, 1.5 MB/s 2020-11-19T02:51:17.524 INFO:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-19T02:51:17.571 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-19T02:51:17.572 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T02:51:17.572 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-11-19T02:51:17.572 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T02:51:17.572 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2020-11-19 02:47:57.627582034 +0000 2020-11-19T02:51:17.573 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2020-11-19 02:47:57.515526035 +0000 2020-11-19T02:51:17.573 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2020-11-19 02:47:57.515526035 +0000 2020-11-19T02:51:17.573 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2020-11-19T02:51:17.580 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2020-11-19T02:51:17.581 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2020-11-19T02:51:17.581 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.00038199 s, 1.3 MB/s 2020-11-19T02:51:17.585 INFO:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-19T02:51:17.635 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-19T02:51:17.636 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T02:51:17.636 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-11-19T02:51:17.636 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T02:51:17.636 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2020-11-19 02:47:57.863700030 +0000 2020-11-19T02:51:17.636 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2020-11-19 02:47:57.775656032 +0000 2020-11-19T02:51:17.637 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2020-11-19 02:47:57.775656032 +0000 2020-11-19T02:51:17.637 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2020-11-19T02:51:17.644 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2020-11-19T02:51:17.644 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2020-11-19T02:51:17.644 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000385962 s, 1.3 MB/s 2020-11-19T02:51:17.648 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T02:51:17.649 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-19T02:51:17.649 INFO:tasks.ceph:Generating config... 2020-11-19T02:51:17.650 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-19T02:51:17.650 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-19T02:51:17.651 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-19T02:51:17.651 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T02:51:17.651 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-19T02:51:17.651 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-19T02:51:17.652 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-19T02:51:17.652 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-19T02:51:17.652 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-19T02:51:17.652 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-19T02:51:17.653 INFO:tasks.ceph:[client] setuser = ceph 2020-11-19T02:51:17.653 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-19T02:51:17.653 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-19T02:51:17.654 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-19T02:51:17.654 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-19T02:51:17.654 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-19T02:51:17.654 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-19T02:51:17.655 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-19T02:51:17.655 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-19T02:51:17.655 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-19T02:51:17.655 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-19T02:51:17.656 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-19T02:51:17.656 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-19T02:51:17.656 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-19T02:51:17.657 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-19T02:51:17.657 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-19T02:51:17.657 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-19T02:51:17.657 INFO:tasks.ceph:Setting up c2.mon.a... 2020-11-19T02:51:17.658 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-11-19T02:51:17.723 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/c2.keyring 2020-11-19T02:51:17.725 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-11-19T02:51:17.803 INFO:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T02:51:17.859 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.160')] 2020-11-19T02:51:17.860 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.160', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-19T02:51:17.861 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:17.865 INFO:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T02:51:17.905 INFO:teuthology.orchestra.run.smithi160.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T02:51:17.907 INFO:teuthology.orchestra.run.smithi160.stderr:6+1 records in 2020-11-19T02:51:17.907 INFO:teuthology.orchestra.run.smithi160.stderr:6+1 records out 2020-11-19T02:51:17.907 INFO:teuthology.orchestra.run.smithi160.stderr:3136 bytes (3.1 kB, 3.1 KiB) copied, 0.000104797 s, 29.9 MB/s 2020-11-19T02:51:17.908 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.160 --print /home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:17.979 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:51:17.980 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:51:17.980 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:17.980 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: generated fsid 9cf7d6fd-5a6e-4fcd-bab6-8458419b847c 2020-11-19T02:51:17.981 INFO:teuthology.orchestra.run.smithi160.stdout:epoch 0 2020-11-19T02:51:17.981 INFO:teuthology.orchestra.run.smithi160.stdout:fsid 9cf7d6fd-5a6e-4fcd-bab6-8458419b847c 2020-11-19T02:51:17.981 INFO:teuthology.orchestra.run.smithi160.stdout:last_changed 2020-11-19T02:51:17.981244+0000 2020-11-19T02:51:17.981 INFO:teuthology.orchestra.run.smithi160.stdout:created 2020-11-19T02:51:17.981244+0000 2020-11-19T02:51:17.981 INFO:teuthology.orchestra.run.smithi160.stdout:min_mon_release 0 (unknown) 2020-11-19T02:51:17.982 INFO:teuthology.orchestra.run.smithi160.stdout:election_strategy: 1 2020-11-19T02:51:17.982 INFO:teuthology.orchestra.run.smithi160.stdout:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2020-11-19T02:51:17.982 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-11-19T02:51:17.983 INFO:teuthology.orchestra.run.smithi160:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T02:51:18.031 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 9cf7d6fd-5a6e-4fcd-bab6-8458419b847c... 2020-11-19T02:51:18.032 INFO:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-19T02:51:18.035 INFO:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-19T02:51:18.076 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2020-11-19T02:51:18.076 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2020-11-19T02:51:18.076 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-19T02:51:18.077 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2020-11-19T02:51:18.077 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.077 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2020-11-19T02:51:18.077 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.078 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2020-11-19T02:51:18.078 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.078 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2020-11-19T02:51:18.078 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2020-11-19T02:51:18.078 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.079 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2020-11-19T02:51:18.079 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2020-11-19T02:51:18.079 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.079 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-19T02:51:18.079 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2020-11-19T02:51:18.080 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.080 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2020-11-19T02:51:18.080 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-19T02:51:18.080 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.080 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2020-11-19T02:51:18.081 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.081 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2020-11-19T02:51:18.081 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2020-11-19T02:51:18.081 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2020-11-19T02:51:18.081 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2020-11-19T02:51:18.082 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2020-11-19T02:51:18.082 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2020-11-19T02:51:18.082 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2020-11-19T02:51:18.082 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-19T02:51:18.082 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-19T02:51:18.083 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2020-11-19T02:51:18.083 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.083 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-19T02:51:18.083 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.083 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2020-11-19T02:51:18.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.084 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2020-11-19T02:51:18.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.084 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2020-11-19T02:51:18.084 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2020-11-19T02:51:18.085 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2020-11-19T02:51:18.085 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = 172.21.15.160 2020-11-19T02:51:18.085 INFO:teuthology.orchestra.run.smithi086.stdout: osd heartbeat grace = 40 2020-11-19T02:51:18.085 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = 9cf7d6fd-5a6e-4fcd-bab6-8458419b847c 2020-11-19T02:51:18.085 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.086 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2020-11-19T02:51:18.086 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2020-11-19T02:51:18.086 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.086 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2020-11-19T02:51:18.086 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2020-11-19T02:51:18.087 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.087 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2020-11-19T02:51:18.087 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2020-11-19T02:51:18.087 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.087 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2020-11-19T02:51:18.088 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2020-11-19T02:51:18.088 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2020-11-19T02:51:18.088 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.088 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2020-11-19T02:51:18.088 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2020-11-19T02:51:18.089 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.089 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2020-11-19T02:51:18.089 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.089 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2020-11-19T02:51:18.089 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.090 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2020-11-19T02:51:18.090 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.090 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2020-11-19T02:51:18.091 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2020-11-19T02:51:18.091 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.092 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2020-11-19T02:51:18.092 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2020-11-19T02:51:18.092 INFO:teuthology.orchestra.run.smithi086.stdout: debug filestore = 20 2020-11-19T02:51:18.092 INFO:teuthology.orchestra.run.smithi086.stdout: debug journal = 20 2020-11-19T02:51:18.092 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 20 2020-11-19T02:51:18.093 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 25 2020-11-19T02:51:18.093 INFO:teuthology.orchestra.run.smithi086.stdout: osd fast shutdown = False 2020-11-19T02:51:18.093 INFO:teuthology.orchestra.run.smithi086.stdout: rocksdb delete range threshold = 0 2020-11-19T02:51:18.093 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.094 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2020-11-19T02:51:18.094 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2020-11-19T02:51:18.094 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2020-11-19T02:51:18.094 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2020-11-19T02:51:18.094 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2020-11-19T02:51:18.095 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2020-11-19T02:51:18.095 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2020-11-19T02:51:18.095 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2020-11-19T02:51:18.096 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.096 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2020-11-19T02:51:18.096 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2020-11-19T02:51:18.096 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2020-11-19T02:51:18.097 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2020-11-19T02:51:18.097 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2020-11-19T02:51:18.097 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2020-11-19T02:51:18.097 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2020-11-19T02:51:18.098 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2020-11-19T02:51:18.098 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2020-11-19T02:51:18.098 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2020-11-19T02:51:18.098 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2020-11-19T02:51:18.098 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd crush smoke test = False 2020-11-19T02:51:18.099 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:51:18.099 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2020-11-19T02:51:18.099 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2020-11-19T02:51:18.099 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2020-11-19T02:51:18.100 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2020-11-19T02:51:18.100 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-19T02:51:18.100 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-19T02:51:18.100 INFO:teuthology.orchestra.run.smithi086.stdout: debug rgw = 20 2020-11-19T02:51:18.101 INFO:teuthology.orchestra.run.smithi086.stdout: rgw crypt require ssl = False 2020-11-19T02:51:18.101 INFO:teuthology.orchestra.run.smithi086.stdout: rgw crypt s3 kms backend = testing 2020-11-19T02:51:18.101 INFO:teuthology.orchestra.run.smithi086.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T02:51:18.102 INFO:teuthology.orchestra.run.smithi086.stdout: rgw curl low speed time = 300 2020-11-19T02:51:18.102 INFO:teuthology.orchestra.run.smithi086.stdout: rgw data log num shards = 4 2020-11-19T02:51:18.102 INFO:teuthology.orchestra.run.smithi086.stdout: rgw md log max shards = 4 2020-11-19T02:51:18.102 INFO:teuthology.orchestra.run.smithi086.stdout: rgw sync log trim interval = 0 2020-11-19T02:51:18.103 INFO:teuthology.orchestra.run.smithi086.stdout: rgw sync obj etag verify = True 2020-11-19T02:51:18.104 INFO:teuthology.orchestra.run.smithi086.stdout: setgroup = ceph 2020-11-19T02:51:18.104 INFO:teuthology.orchestra.run.smithi086.stdout: setuser = ceph 2020-11-19T02:51:18.104 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2020-11-19T02:51:18.104 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2020-11-19T02:51:18.104 INFO:teuthology.orchestra.run.smithi086.stdout: osd_max_omap_entries_per_request = 10 2020-11-19T02:51:18.105 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2020-11-19T02:51:18.105 INFO:teuthology.orchestra.run.smithi086.stdout: osd_max_omap_entries_per_request = 1000 2020-11-19T02:51:18.105 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2020-11-19T02:51:18.106 INFO:teuthology.orchestra.run.smithi086.stdout: osd_max_omap_entries_per_request = 10000 2020-11-19T02:51:18.107 INFO:teuthology.orchestra.run.smithi160.stdout:[global] 2020-11-19T02:51:18.107 INFO:teuthology.orchestra.run.smithi160.stdout: chdir = "" 2020-11-19T02:51:18.108 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-19T02:51:18.108 INFO:teuthology.orchestra.run.smithi160.stdout: auth supported = cephx 2020-11-19T02:51:18.108 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.109 INFO:teuthology.orchestra.run.smithi160.stdout: filestore xattr use omap = true 2020-11-19T02:51:18.109 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.109 INFO:teuthology.orchestra.run.smithi160.stdout: mon clock drift allowed = 1.000 2020-11-19T02:51:18.109 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.110 INFO:teuthology.orchestra.run.smithi160.stdout: osd crush chooseleaf type = 0 2020-11-19T02:51:18.110 INFO:teuthology.orchestra.run.smithi160.stdout: auth debug = true 2020-11-19T02:51:18.110 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.110 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on old message = true 2020-11-19T02:51:18.110 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on bug = true 2020-11-19T02:51:18.111 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.111 INFO:teuthology.orchestra.run.smithi160.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-19T02:51:18.111 INFO:teuthology.orchestra.run.smithi160.stdout: mon pg warn max object skew = 0 2020-11-19T02:51:18.111 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.111 INFO:teuthology.orchestra.run.smithi160.stdout: # disable pg_autoscaler by default for new pools 2020-11-19T02:51:18.112 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-19T02:51:18.112 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.112 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default size = 2 2020-11-19T02:51:18.112 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.113 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow primary affinity = true 2020-11-19T02:51:18.113 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow pg remap = true 2020-11-19T02:51:18.113 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on legacy crush tunables = false 2020-11-19T02:51:18.114 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on crush straw calc version zero = false 2020-11-19T02:51:18.114 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on no sortbitwise = false 2020-11-19T02:51:18.114 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on osd down out interval zero = false 2020-11-19T02:51:18.114 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on too few osds = false 2020-11-19T02:51:18.115 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-19T02:51:18.115 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-19T02:51:18.115 INFO:teuthology.orchestra.run.smithi160.stdout: mon_allow_pool_size_one = true 2020-11-19T02:51:18.115 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.115 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-19T02:51:18.116 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.116 INFO:teuthology.orchestra.run.smithi160.stdout: osd default data pool replay window = 5 2020-11-19T02:51:18.116 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.116 INFO:teuthology.orchestra.run.smithi160.stdout: mon allow pool delete = true 2020-11-19T02:51:18.117 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.117 INFO:teuthology.orchestra.run.smithi160.stdout: mon cluster log file level = debug 2020-11-19T02:51:18.117 INFO:teuthology.orchestra.run.smithi160.stdout: debug asserts on shutdown = true 2020-11-19T02:51:18.117 INFO:teuthology.orchestra.run.smithi160.stdout: mon health detail to clog = false 2020-11-19T02:51:18.118 INFO:teuthology.orchestra.run.smithi160.stdout: mon host = 172.21.15.160 2020-11-19T02:51:18.118 INFO:teuthology.orchestra.run.smithi160.stdout: osd heartbeat grace = 40 2020-11-19T02:51:18.118 INFO:teuthology.orchestra.run.smithi160.stdout: fsid = 9cf7d6fd-5a6e-4fcd-bab6-8458419b847c 2020-11-19T02:51:18.118 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.119 INFO:teuthology.orchestra.run.smithi160.stdout:[osd] 2020-11-19T02:51:18.119 INFO:teuthology.orchestra.run.smithi160.stdout: osd journal size = 100 2020-11-19T02:51:18.119 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.119 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub load threshold = 5.0 2020-11-19T02:51:18.120 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub max interval = 600 2020-11-19T02:51:18.120 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.120 INFO:teuthology.orchestra.run.smithi160.stdout: osd recover clone overlap = true 2020-11-19T02:51:18.120 INFO:teuthology.orchestra.run.smithi160.stdout: osd recovery max chunk = 1048576 2020-11-19T02:51:18.121 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.121 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug shutdown = true 2020-11-19T02:51:18.121 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug op order = true 2020-11-19T02:51:18.121 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify stray on activate = true 2020-11-19T02:51:18.121 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.122 INFO:teuthology.orchestra.run.smithi160.stdout: osd open classes on start = true 2020-11-19T02:51:18.122 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug pg log writeout = true 2020-11-19T02:51:18.122 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.122 INFO:teuthology.orchestra.run.smithi160.stdout: osd deep scrub update digest min age = 30 2020-11-19T02:51:18.123 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.123 INFO:teuthology.orchestra.run.smithi160.stdout: osd map max advance = 10 2020-11-19T02:51:18.123 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.123 INFO:teuthology.orchestra.run.smithi160.stdout: journal zero on create = true 2020-11-19T02:51:18.124 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.124 INFO:teuthology.orchestra.run.smithi160.stdout: filestore ondisk finisher threads = 3 2020-11-19T02:51:18.124 INFO:teuthology.orchestra.run.smithi160.stdout: filestore apply finisher threads = 3 2020-11-19T02:51:18.124 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.125 INFO:teuthology.orchestra.run.smithi160.stdout: bdev debug aio = true 2020-11-19T02:51:18.125 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug misdirected ops = true 2020-11-19T02:51:18.125 INFO:teuthology.orchestra.run.smithi160.stdout: debug filestore = 20 2020-11-19T02:51:18.125 INFO:teuthology.orchestra.run.smithi160.stdout: debug journal = 20 2020-11-19T02:51:18.126 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 20 2020-11-19T02:51:18.126 INFO:teuthology.orchestra.run.smithi160.stdout: debug osd = 25 2020-11-19T02:51:18.126 INFO:teuthology.orchestra.run.smithi160.stdout: osd fast shutdown = False 2020-11-19T02:51:18.126 INFO:teuthology.orchestra.run.smithi160.stdout: rocksdb delete range threshold = 0 2020-11-19T02:51:18.126 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.127 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr] 2020-11-19T02:51:18.127 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2020-11-19T02:51:18.127 INFO:teuthology.orchestra.run.smithi160.stdout: debug mgr = 20 2020-11-19T02:51:18.127 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2020-11-19T02:51:18.128 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2020-11-19T02:51:18.128 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2020-11-19T02:51:18.128 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2020-11-19T02:51:18.128 INFO:teuthology.orchestra.run.smithi160.stdout: mgr/telemetry/nag = false 2020-11-19T02:51:18.128 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.129 INFO:teuthology.orchestra.run.smithi160.stdout:[mon] 2020-11-19T02:51:18.129 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2020-11-19T02:51:18.129 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2020-11-19T02:51:18.129 INFO:teuthology.orchestra.run.smithi160.stdout: debug paxos = 20 2020-11-19T02:51:18.130 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2020-11-19T02:51:18.130 INFO:teuthology.orchestra.run.smithi160.stdout: mon data avail warn = 5 2020-11-19T02:51:18.130 INFO:teuthology.orchestra.run.smithi160.stdout: mon mgr mkfs grace = 240 2020-11-19T02:51:18.131 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2020-11-19T02:51:18.131 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd reporter subtree level = osd 2020-11-19T02:51:18.131 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd prime pg temp = true 2020-11-19T02:51:18.131 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2020-11-19T02:51:18.131 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd crush smoke test = False 2020-11-19T02:51:18.132 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:18.132 INFO:teuthology.orchestra.run.smithi160.stdout:[client] 2020-11-19T02:51:18.132 INFO:teuthology.orchestra.run.smithi160.stdout: rgw cache enabled = true 2020-11-19T02:51:18.132 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable ops log = true 2020-11-19T02:51:18.132 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable usage log = true 2020-11-19T02:51:18.133 INFO:teuthology.orchestra.run.smithi160.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-19T02:51:18.133 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-19T02:51:18.133 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw = 20 2020-11-19T02:51:18.133 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt require ssl = False 2020-11-19T02:51:18.133 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms backend = testing 2020-11-19T02:51:18.134 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T02:51:18.134 INFO:teuthology.orchestra.run.smithi160.stdout: rgw curl low speed time = 300 2020-11-19T02:51:18.135 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data log num shards = 4 2020-11-19T02:51:18.136 INFO:teuthology.orchestra.run.smithi160.stdout: rgw md log max shards = 4 2020-11-19T02:51:18.136 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync log trim interval = 0 2020-11-19T02:51:18.136 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync obj etag verify = True 2020-11-19T02:51:18.137 INFO:teuthology.orchestra.run.smithi160.stdout: setgroup = ceph 2020-11-19T02:51:18.137 INFO:teuthology.orchestra.run.smithi160.stdout: setuser = ceph 2020-11-19T02:51:18.137 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.a] 2020-11-19T02:51:18.138 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2020-11-19T02:51:18.138 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10 2020-11-19T02:51:18.139 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2020-11-19T02:51:18.139 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 1000 2020-11-19T02:51:18.139 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2020-11-19T02:51:18.140 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10000 2020-11-19T02:51:18.141 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-11-19T02:51:18.142 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-11-19T02:51:18.231 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-19T02:51:18.231 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:18.231 INFO:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-19T02:51:18.236 INFO:teuthology.orchestra.run.smithi160.stderr:+ dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-19T02:51:18.238 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.] 2020-11-19T02:51:18.238 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCl3bVfzbbNLxAAF+44BhOxWnw82XB8hW794Q== 2020-11-19T02:51:18.238 INFO:teuthology.orchestra.run.smithi160.stdout:[client.admin] 2020-11-19T02:51:18.239 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCm3bVfZTq2DRAAIMZibMXYjJsTvkykNqCk1A== 2020-11-19T02:51:18.239 INFO:teuthology.orchestra.run.smithi160.stdout: caps mds = "allow *" 2020-11-19T02:51:18.239 INFO:teuthology.orchestra.run.smithi160.stdout: caps mgr = "allow *" 2020-11-19T02:51:18.239 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "allow *" 2020-11-19T02:51:18.240 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "allow *" 2020-11-19T02:51:18.241 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:18.241 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:18.241 INFO:teuthology.orchestra.run.smithi160.stderr:206 bytes copied, 0.00015845 s, 1.3 MB/s 2020-11-19T02:51:18.242 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:18.242 INFO:teuthology.orchestra.run.smithi160:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-19T02:51:18.285 INFO:teuthology.orchestra.run.smithi160.stderr:+ dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-19T02:51:18.286 INFO:teuthology.orchestra.run.smithi160.stdout: �����ZnOͺ��XA��|�ݵ_��|:�ݵ_��|:�aXa �����a 2020-11-19T02:51:18.287 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:18.287 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:18.288 INFO:teuthology.orchestra.run.smithi160.stderr:201 bytes copied, 0.00013298 s, 1.5 MB/s 2020-11-19T02:51:18.288 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2020-11-19T02:51:18.288 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:51:18.289 INFO:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/c2.keyring 2020-11-19T02:51:18.289 INFO:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T02:51:18.295 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-19T02:51:18.304 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:51:18.305 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:51:18.305 INFO:teuthology.orchestra.run.smithi086.stderr:206 bytes copied, 7.0244e-05 s, 2.9 MB/s 2020-11-19T02:51:18.305 INFO:teuthology.orchestra.run.smithi086.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T02:51:18.314 INFO:teuthology.orchestra.run.smithi086:> set -ex 2020-11-19T02:51:18.314 INFO:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:18.360 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:18.362 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2020-11-19T02:51:18.363 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2020-11-19T02:51:18.363 INFO:teuthology.orchestra.run.smithi086.stderr:201 bytes copied, 6.866e-05 s, 2.9 MB/s 2020-11-19T02:51:18.364 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi160.front.sepia.ceph.com 2020-11-19T02:51:18.365 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:18.365 INFO:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/c2.keyring 2020-11-19T02:51:18.366 INFO:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T02:51:18.371 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-19T02:51:18.381 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:18.381 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:18.382 INFO:teuthology.orchestra.run.smithi160.stderr:206 bytes copied, 0.000136937 s, 1.5 MB/s 2020-11-19T02:51:18.383 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T02:51:18.391 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:18.392 INFO:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:18.437 INFO:teuthology.orchestra.run.smithi160.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:18.439 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:18.439 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:18.439 INFO:teuthology.orchestra.run.smithi160.stderr:201 bytes copied, 0.000131783 s, 1.5 MB/s 2020-11-19T02:51:18.439 INFO:tasks.ceph:Setting up mon nodes... 2020-11-19T02:51:18.440 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-19T02:51:18.440 INFO:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-11-19T02:51:18.524 INFO:teuthology.orchestra.run.smithi160.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-11-19T02:51:18.527 INFO:tasks.ceph:Setting up mds nodes... 2020-11-19T02:51:18.527 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-19T02:51:18.528 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-11-19T02:51:18.613 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/c2.client.0.keyring 2020-11-19T02:51:18.625 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-11-19T02:51:18.708 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/c2.client.1.keyring 2020-11-19T02:51:18.720 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-19T02:51:18.721 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-19T02:51:18.721 INFO:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-11-19T02:51:18.782 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T02:51:18.783 INFO:tasks.ceph:role: c2.osd.0 2020-11-19T02:51:18.783 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com 2020-11-19T02:51:18.783 INFO:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-19T02:51:19.337 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T02:51:19.337 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T02:51:19.337 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T02:51:19.338 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T02:51:19.338 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2020-11-19T02:51:19.338 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T02:51:19.338 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T02:51:19.338 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T02:51:19.339 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T02:51:19.340 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2020-11-19T02:51:19.340 INFO:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-11-19T02:51:19.424 INFO:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-11-19T02:51:19.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T02:51:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T02:51:19.439 INFO:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-11-19T02:51:19.496 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T02:51:19.528 INFO:tasks.ceph:role: c2.osd.1 2020-11-19T02:51:19.531 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com 2020-11-19T02:51:19.533 INFO:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-19T02:51:20.019 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T02:51:20.019 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T02:51:20.020 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T02:51:20.020 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T02:51:20.020 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2020-11-19T02:51:20.020 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T02:51:20.020 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T02:51:20.021 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T02:51:20.021 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T02:51:20.022 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2020-11-19T02:51:20.022 INFO:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-11-19T02:51:20.065 INFO:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-11-19T02:51:20.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T02:51:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T02:51:20.122 INFO:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-11-19T02:51:20.179 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T02:51:20.179 INFO:tasks.ceph:role: c2.osd.2 2020-11-19T02:51:20.179 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com 2020-11-19T02:51:20.180 INFO:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-19T02:51:20.693 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T02:51:20.693 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T02:51:20.694 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T02:51:20.694 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T02:51:20.694 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2020-11-19T02:51:20.694 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T02:51:20.694 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T02:51:20.695 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T02:51:20.695 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T02:51:20.696 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2020-11-19T02:51:20.696 INFO:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-11-19T02:51:20.755 INFO:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-11-19T02:51:20.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T02:51:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T02:51:20.772 INFO:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:20.870 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:20.867+0000 7f24247c2d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-11-19T02:51:20.874 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:20.867+0000 7f24247c2d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-11-19T02:51:20.875 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:20.867+0000 7f24247c2d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-19T02:51:20.876 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:20.867+0000 7f24247c2d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-19T02:51:20.876 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:20.867+0000 7f24247c2d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-19T02:51:20.877 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:20.867+0000 7f24247c2d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-11-19T02:51:23.049 INFO:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-11-19T02:51:23.065 INFO:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:23.147 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:23.143+0000 7ff6d8b8ad80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-11-19T02:51:23.147 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:23.143+0000 7ff6d8b8ad80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-11-19T02:51:23.148 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:23.143+0000 7ff6d8b8ad80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-19T02:51:23.148 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:23.143+0000 7ff6d8b8ad80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-19T02:51:23.148 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:23.143+0000 7ff6d8b8ad80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-19T02:51:23.148 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:23.143+0000 7ff6d8b8ad80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-11-19T02:51:25.317 INFO:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-11-19T02:51:25.395 INFO:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:25.482 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:25.479+0000 7fcc909f7d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-11-19T02:51:25.483 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:25.479+0000 7fcc909f7d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-11-19T02:51:25.483 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:25.479+0000 7fcc909f7d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-19T02:51:25.483 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:25.479+0000 7fcc909f7d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-19T02:51:25.484 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:25.479+0000 7fcc909f7d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-19T02:51:25.489 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:51:25.479+0000 7fcc909f7d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-11-19T02:51:27.644 INFO:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-11-19T02:51:27.700 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-19T02:51:27.701 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:27.701 INFO:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-19T02:51:27.745 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-19T02:51:27.754 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr.x] 2020-11-19T02:51:27.755 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCm3bVf44BiHxAA+IW+cMjfTvQ4Yezv00CdPw== 2020-11-19T02:51:27.755 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:27.755 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:27.756 INFO:teuthology.orchestra.run.smithi160.stderr:56 bytes copied, 0.000210403 s, 266 kB/s 2020-11-19T02:51:27.756 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:27.757 INFO:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-19T02:51:27.802 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-19T02:51:27.811 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2020-11-19T02:51:27.811 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCo3bVfLcHyMxAA5ZcX7hhuMvJd99ftXS5aZg== 2020-11-19T02:51:27.811 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:27.812 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:27.812 INFO:teuthology.orchestra.run.smithi160.stderr:56 bytes copied, 0.000197724 s, 283 kB/s 2020-11-19T02:51:27.813 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:27.813 INFO:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-19T02:51:27.861 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-19T02:51:27.870 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2020-11-19T02:51:27.870 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCr3bVfuPbbCBAAQ0shF8M5GckJsxtVqPO5ug== 2020-11-19T02:51:27.871 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:27.871 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:27.871 INFO:teuthology.orchestra.run.smithi160.stderr:56 bytes copied, 0.00020064 s, 279 kB/s 2020-11-19T02:51:27.872 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:27.872 INFO:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-19T02:51:27.922 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-19T02:51:27.929 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2020-11-19T02:51:27.929 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCt3bVfB4XZHBAAZFR0vfm9fHHKBmW8FBjq+Q== 2020-11-19T02:51:27.930 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:27.930 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:27.930 INFO:teuthology.orchestra.run.smithi160.stderr:56 bytes copied, 0.00019988 s, 280 kB/s 2020-11-19T02:51:27.931 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:27.932 INFO:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-19T02:51:27.978 INFO:teuthology.orchestra.run.smithi160.stderr:+ dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-19T02:51:27.979 INFO:teuthology.orchestra.run.smithi160.stdout:[client.0] 2020-11-19T02:51:27.979 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCm3bVfgjOlJBAAobebvEAzTyWZmerqfZvYrg== 2020-11-19T02:51:27.980 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:27.980 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:27.980 INFO:teuthology.orchestra.run.smithi160.stderr:59 bytes copied, 0.000164357 s, 359 kB/s 2020-11-19T02:51:27.980 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:27.981 INFO:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-19T02:51:28.027 INFO:teuthology.orchestra.run.smithi160.stderr:+ dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-19T02:51:28.027 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:28.027 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:28.028 INFO:teuthology.orchestra.run.smithi160.stderr:59 bytes copied, 0.0001521 s, 388 kB/s 2020-11-19T02:51:28.028 INFO:teuthology.orchestra.run.smithi160.stdout:[client.1] 2020-11-19T02:51:28.028 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCm3bVfY/ZYKhAAUaAznvCT6lj1kSFMf1MRiQ== 2020-11-19T02:51:28.029 INFO:tasks.ceph:Adding keys to all mons... 2020-11-19T02:51:28.029 INFO:teuthology.orchestra.run.smithi160:> sudo tee -a /etc/ceph/c2.keyring 2020-11-19T02:51:28.082 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr.x] 2020-11-19T02:51:28.082 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCm3bVf44BiHxAA+IW+cMjfTvQ4Yezv00CdPw== 2020-11-19T02:51:28.082 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2020-11-19T02:51:28.083 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCo3bVfLcHyMxAA5ZcX7hhuMvJd99ftXS5aZg== 2020-11-19T02:51:28.083 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2020-11-19T02:51:28.083 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCr3bVfuPbbCBAAQ0shF8M5GckJsxtVqPO5ug== 2020-11-19T02:51:28.083 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2020-11-19T02:51:28.083 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCt3bVfB4XZHBAAZFR0vfm9fHHKBmW8FBjq+Q== 2020-11-19T02:51:28.084 INFO:teuthology.orchestra.run.smithi160.stdout:[client.0] 2020-11-19T02:51:28.084 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCm3bVfgjOlJBAAobebvEAzTyWZmerqfZvYrg== 2020-11-19T02:51:28.084 INFO:teuthology.orchestra.run.smithi160.stdout:[client.1] 2020-11-19T02:51:28.084 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCm3bVfY/ZYKhAAUaAznvCT6lj1kSFMf1MRiQ== 2020-11-19T02:51:28.085 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-19T02:51:28.172 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T02:51:28.255 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T02:51:28.339 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T02:51:28.426 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-19T02:51:28.515 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-19T02:51:28.613 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-19T02:51:28.613 INFO:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-11-19T02:51:28.679 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-11-19T02:51:28.788 INFO:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-11-19T02:51:28.809 INFO:teuthology.orchestra.run.smithi160:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-11-19T02:51:28.861 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-11-19T02:51:28.862 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-11-19T02:51:28.862 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-11-19T02:51:28.906 INFO:tasks.ceph.c2.mon.a:Started 2020-11-19T02:51:28.907 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-11-19T02:51:28.907 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-11-19T02:51:28.907 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-11-19T02:51:28.911 INFO:tasks.ceph.c2.mgr.x:Started 2020-11-19T02:51:28.912 INFO:tasks.ceph:Setting crush tunables to default 2020-11-19T02:51:28.912 INFO:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd crush tunables default 2020-11-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi160.stderr:adjusted tunables profile to default 2020-11-19T02:51:29.251 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-11-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi160.stdout:259a083f-80bc-4873-a52a-1c96c2b85c5a 2020-11-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi160.stderr:37 bytes copied, 0.00015287 s, 242 kB/s 2020-11-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi160.stdout:83c3c6d9-543b-4b3a-8678-0bf07ff1d57f 2020-11-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi160.stderr:37 bytes copied, 0.000126887 s, 292 kB/s 2020-11-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi160:> set -ex 2020-11-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-19T02:51:29.369 INFO:teuthology.orchestra.run.smithi160.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-19T02:51:29.377 INFO:teuthology.orchestra.run.smithi160.stdout:1a98cba8-dd5f-4281-aa3a-d45386cf1f74 2020-11-19T02:51:29.377 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2020-11-19T02:51:29.377 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2020-11-19T02:51:29.378 INFO:teuthology.orchestra.run.smithi160.stderr:37 bytes copied, 0.000166327 s, 222 kB/s 2020-11-19T02:51:29.378 INFO:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd new 259a083f-80bc-4873-a52a-1c96c2b85c5a 0 2020-11-19T02:51:29.638 INFO:teuthology.orchestra.run.smithi160.stdout:0 2020-11-19T02:51:29.652 INFO:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd new 83c3c6d9-543b-4b3a-8678-0bf07ff1d57f 1 2020-11-19T02:51:29.876 INFO:teuthology.orchestra.run.smithi160.stdout:1 2020-11-19T02:51:29.888 INFO:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd new 1a98cba8-dd5f-4281-aa3a-d45386cf1f74 2 2020-11-19T02:51:30.103 INFO:teuthology.orchestra.run.smithi160.stdout:2 2020-11-19T02:51:30.117 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-11-19T02:51:30.117 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-11-19T02:51:30.119 INFO:tasks.ceph.c2.osd.0:Started 2020-11-19T02:51:30.120 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-11-19T02:51:30.120 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-11-19T02:51:30.122 INFO:tasks.ceph.c2.osd.1:Started 2020-11-19T02:51:30.123 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-11-19T02:51:30.123 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-11-19T02:51:30.125 INFO:tasks.ceph.c2.osd.2:Started 2020-11-19T02:51:30.125 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-19T02:51:30.429 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-19T02:51:30.658 INFO:teuthology.misc.health.smithi160.stdout: 2020-11-19T02:51:30.658 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":5,"fsid":"9cf7d6fd-5a6e-4fcd-bab6-8458419b847c","created":"2020-11-19T02:51:29.021642+0000","modified":"2020-11-19T02:51:30.102023+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"259a083f-80bc-4873-a52a-1c96c2b85c5a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"83c3c6d9-543b-4b3a-8678-0bf07ff1d57f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1a98cba8-dd5f-4281-aa3a-d45386cf1f74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:30.666 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-19T02:51:30.744 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2020-11-19T02:51:30.739+0000 7ff7cb6aed80 -1 Falling back to public interface 2020-11-19T02:51:30.747 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2020-11-19T02:51:30.739+0000 7f42e5c8ed80 -1 Falling back to public interface 2020-11-19T02:51:30.748 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2020-11-19T02:51:30.739+0000 7f47bd860d80 -1 Falling back to public interface 2020-11-19T02:51:31.127 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2020-11-19T02:51:31.123+0000 7f47bd860d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-19T02:51:31.353 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2020-11-19T02:51:31.351+0000 7ff7cb6aed80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-19T02:51:31.371 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2020-11-19T02:51:31.367+0000 7f42e5c8ed80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-19T02:51:33.051 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2020-11-19T02:51:33.047+0000 7ff7c2b78700 -1 osd.2 0 waiting for initial osdmap 2020-11-19T02:51:33.052 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2020-11-19T02:51:33.047+0000 7f42dd158700 -1 osd.1 0 waiting for initial osdmap 2020-11-19T02:51:33.053 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2020-11-19T02:51:33.047+0000 7f47b4d2a700 -1 osd.0 0 waiting for initial osdmap 2020-11-19T02:51:36.969 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-19T02:51:37.256 INFO:teuthology.misc.health.smithi160.stdout: 2020-11-19T02:51:37.257 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":10,"fsid":"9cf7d6fd-5a6e-4fcd-bab6-8458419b847c","created":"2020-11-19T02:51:29.021642+0000","modified":"2020-11-19T02:51:36.064316+0000","last_up_change":"2020-11-19T02:51:34.046435+0000","last_in_change":"2020-11-19T02:51:34.046435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T02:51:32.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"259a083f-80bc-4873-a52a-1c96c2b85c5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":11198},{"type":"v1","addr":"172.21.15.160:6809","nonce":11198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":11198},{"type":"v1","addr":"172.21.15.160:6815","nonce":11198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6821","nonce":11198},{"type":"v1","addr":"172.21.15.160:6823","nonce":11198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6817","nonce":11198},{"type":"v1","addr":"172.21.15.160:6819","nonce":11198}]},"public_addr":"172.21.15.160:6809/11198","cluster_addr":"172.21.15.160:6815/11198","heartbeat_back_addr":"172.21.15.160:6823/11198","heartbeat_front_addr":"172.21.15.160:6819/11198","state":["exists","up"]},{"osd":1,"uuid":"83c3c6d9-543b-4b3a-8678-0bf07ff1d57f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":11199},{"type":"v1","addr":"172.21.15.160:6807","nonce":11199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6811","nonce":11199},{"type":"v1","addr":"172.21.15.160:6813","nonce":11199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":11199},{"type":"v1","addr":"172.21.15.160:6822","nonce":11199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":11199},{"type":"v1","addr":"172.21.15.160:6818","nonce":11199}]},"public_addr":"172.21.15.160:6807/11199","cluster_addr":"172.21.15.160:6813/11199","heartbeat_back_addr":"172.21.15.160:6822/11199","heartbeat_front_addr":"172.21.15.160:6818/11199","state":["exists","up"]},{"osd":2,"uuid":"1a98cba8-dd5f-4281-aa3a-d45386cf1f74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":11201},{"type":"v1","addr":"172.21.15.160:6801","nonce":11201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":11201},{"type":"v1","addr":"172.21.15.160:6803","nonce":11201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":11201},{"type":"v1","addr":"172.21.15.160:6814","nonce":11201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6805","nonce":11201},{"type":"v1","addr":"172.21.15.160:6808","nonce":11201}]},"public_addr":"172.21.15.160:6801/11201","cluster_addr":"172.21.15.160:6803/11201","heartbeat_back_addr":"172.21.15.160:6814/11201","heartbeat_front_addr":"172.21.15.160:6808/11201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.127403+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.413440+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.384598+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:37.270 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-19T02:51:37.271 INFO:tasks.ceph:Creating RBD pool 2020-11-19T02:51:37.271 INFO:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd pool create rbd 8 2020-11-19T02:51:38.096 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'rbd' created 2020-11-19T02:51:38.110 INFO:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-19T02:51:39.102 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-19T02:51:39.115 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-11-19T02:51:39.116 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-19T02:51:39.116 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-19T02:51:39.397 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:39.398 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":12,"fsid":"9cf7d6fd-5a6e-4fcd-bab6-8458419b847c","created":"2020-11-19T02:51:29.021642+0000","modified":"2020-11-19T02:51:39.080709+0000","last_up_change":"2020-11-19T02:51:34.046435+0000","last_in_change":"2020-11-19T02:51:34.046435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T02:51:32.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T02:51:37.562102+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"259a083f-80bc-4873-a52a-1c96c2b85c5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":11198},{"type":"v1","addr":"172.21.15.160:6809","nonce":11198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":11198},{"type":"v1","addr":"172.21.15.160:6815","nonce":11198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6821","nonce":11198},{"type":"v1","addr":"172.21.15.160:6823","nonce":11198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6817","nonce":11198},{"type":"v1","addr":"172.21.15.160:6819","nonce":11198}]},"public_addr":"172.21.15.160:6809/11198","cluster_addr":"172.21.15.160:6815/11198","heartbeat_back_addr":"172.21.15.160:6823/11198","heartbeat_front_addr":"172.21.15.160:6819/11198","state":["exists","up"]},{"osd":1,"uuid":"83c3c6d9-543b-4b3a-8678-0bf07ff1d57f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":11199},{"type":"v1","addr":"172.21.15.160:6807","nonce":11199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6811","nonce":11199},{"type":"v1","addr":"172.21.15.160:6813","nonce":11199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":11199},{"type":"v1","addr":"172.21.15.160:6822","nonce":11199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":11199},{"type":"v1","addr":"172.21.15.160:6818","nonce":11199}]},"public_addr":"172.21.15.160:6807/11199","cluster_addr":"172.21.15.160:6813/11199","heartbeat_back_addr":"172.21.15.160:6822/11199","heartbeat_front_addr":"172.21.15.160:6818/11199","state":["exists","up"]},{"osd":2,"uuid":"1a98cba8-dd5f-4281-aa3a-d45386cf1f74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":11201},{"type":"v1","addr":"172.21.15.160:6801","nonce":11201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":11201},{"type":"v1","addr":"172.21.15.160:6803","nonce":11201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":11201},{"type":"v1","addr":"172.21.15.160:6814","nonce":11201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6805","nonce":11201},{"type":"v1","addr":"172.21.15.160:6808","nonce":11201}]},"public_addr":"172.21.15.160:6801/11201","cluster_addr":"172.21.15.160:6803/11201","heartbeat_back_addr":"172.21.15.160:6814/11201","heartbeat_front_addr":"172.21.15.160:6808/11201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.127403+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.413440+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.384598+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:39.411 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-19T02:51:32.925911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-19T02:51:37.562102+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-19T02:51:39.412 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2020-11-19T02:51:39.706 INFO:teuthology.orchestra.run.smithi160.stdout:pg_num: 1 2020-11-19T02:51:39.720 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2020-11-19T02:51:40.018 INFO:teuthology.orchestra.run.smithi160.stdout:pg_num: 8 2020-11-19T02:51:40.032 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-11-19T02:51:40.033 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-11-19T02:51:40.033 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2020-11-19T02:51:40.359 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:40.371 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":11016},{"type":"v1","addr":"172.21.15.160:6825","nonce":11016}]},"active_addr":"172.21.15.160:6825/11016","active_change":"2020-11-19T02:51:32.897043+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":1906491361}]},{"addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":3731168514}]},{"addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":1420830162}]}]}} 2020-11-19T02:51:40.375 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-11-19T02:51:40.375 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-11-19T02:51:40.376 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-19T02:51:40.665 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:40.665 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":13,"fsid":"9cf7d6fd-5a6e-4fcd-bab6-8458419b847c","created":"2020-11-19T02:51:29.021642+0000","modified":"2020-11-19T02:51:40.094847+0000","last_up_change":"2020-11-19T02:51:34.046435+0000","last_in_change":"2020-11-19T02:51:34.046435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T02:51:32.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T02:51:37.562102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"259a083f-80bc-4873-a52a-1c96c2b85c5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":11198},{"type":"v1","addr":"172.21.15.160:6809","nonce":11198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":11198},{"type":"v1","addr":"172.21.15.160:6815","nonce":11198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6821","nonce":11198},{"type":"v1","addr":"172.21.15.160:6823","nonce":11198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6817","nonce":11198},{"type":"v1","addr":"172.21.15.160:6819","nonce":11198}]},"public_addr":"172.21.15.160:6809/11198","cluster_addr":"172.21.15.160:6815/11198","heartbeat_back_addr":"172.21.15.160:6823/11198","heartbeat_front_addr":"172.21.15.160:6819/11198","state":["exists","up"]},{"osd":1,"uuid":"83c3c6d9-543b-4b3a-8678-0bf07ff1d57f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":11199},{"type":"v1","addr":"172.21.15.160:6807","nonce":11199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6811","nonce":11199},{"type":"v1","addr":"172.21.15.160:6813","nonce":11199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":11199},{"type":"v1","addr":"172.21.15.160:6822","nonce":11199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":11199},{"type":"v1","addr":"172.21.15.160:6818","nonce":11199}]},"public_addr":"172.21.15.160:6807/11199","cluster_addr":"172.21.15.160:6813/11199","heartbeat_back_addr":"172.21.15.160:6822/11199","heartbeat_front_addr":"172.21.15.160:6818/11199","state":["exists","up"]},{"osd":2,"uuid":"1a98cba8-dd5f-4281-aa3a-d45386cf1f74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":11201},{"type":"v1","addr":"172.21.15.160:6801","nonce":11201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":11201},{"type":"v1","addr":"172.21.15.160:6803","nonce":11201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":11201},{"type":"v1","addr":"172.21.15.160:6814","nonce":11201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6805","nonce":11201},{"type":"v1","addr":"172.21.15.160:6808","nonce":11201}]},"public_addr":"172.21.15.160:6801/11201","cluster_addr":"172.21.15.160:6803/11201","heartbeat_back_addr":"172.21.15.160:6814/11201","heartbeat_front_addr":"172.21.15.160:6808/11201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.127403+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.413440+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.384598+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:40.685 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-11-19T02:51:40.686 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-19T02:51:40.982 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:41.733 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":13,"fsid":"9cf7d6fd-5a6e-4fcd-bab6-8458419b847c","created":"2020-11-19T02:51:29.021642+0000","modified":"2020-11-19T02:51:40.094847+0000","last_up_change":"2020-11-19T02:51:34.046435+0000","last_in_change":"2020-11-19T02:51:34.046435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T02:51:32.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T02:51:37.562102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"259a083f-80bc-4873-a52a-1c96c2b85c5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":11198},{"type":"v1","addr":"172.21.15.160:6809","nonce":11198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":11198},{"type":"v1","addr":"172.21.15.160:6815","nonce":11198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6821","nonce":11198},{"type":"v1","addr":"172.21.15.160:6823","nonce":11198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6817","nonce":11198},{"type":"v1","addr":"172.21.15.160:6819","nonce":11198}]},"public_addr":"172.21.15.160:6809/11198","cluster_addr":"172.21.15.160:6815/11198","heartbeat_back_addr":"172.21.15.160:6823/11198","heartbeat_front_addr":"172.21.15.160:6819/11198","state":["exists","up"]},{"osd":1,"uuid":"83c3c6d9-543b-4b3a-8678-0bf07ff1d57f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":11199},{"type":"v1","addr":"172.21.15.160:6807","nonce":11199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6811","nonce":11199},{"type":"v1","addr":"172.21.15.160:6813","nonce":11199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":11199},{"type":"v1","addr":"172.21.15.160:6822","nonce":11199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":11199},{"type":"v1","addr":"172.21.15.160:6818","nonce":11199}]},"public_addr":"172.21.15.160:6807/11199","cluster_addr":"172.21.15.160:6813/11199","heartbeat_back_addr":"172.21.15.160:6822/11199","heartbeat_front_addr":"172.21.15.160:6818/11199","state":["exists","up"]},{"osd":2,"uuid":"1a98cba8-dd5f-4281-aa3a-d45386cf1f74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":11201},{"type":"v1","addr":"172.21.15.160:6801","nonce":11201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":11201},{"type":"v1","addr":"172.21.15.160:6803","nonce":11201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":11201},{"type":"v1","addr":"172.21.15.160:6814","nonce":11201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6805","nonce":11201},{"type":"v1","addr":"172.21.15.160:6808","nonce":11201}]},"public_addr":"172.21.15.160:6801/11201","cluster_addr":"172.21.15.160:6803/11201","heartbeat_back_addr":"172.21.15.160:6814/11201","heartbeat_front_addr":"172.21.15.160:6808/11201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.127403+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.413440+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T02:51:32.384598+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T02:51:41.735 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2020-11-19T02:51:41.934 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2020-11-19T02:51:41.935 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2020-11-19T02:51:42.094 INFO:teuthology.orchestra.run.smithi160.stdout:34359738371 2020-11-19T02:51:42.094 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2020-11-19T02:51:42.266 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2020-11-19T02:51:42.266 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-19T02:51:42.613 INFO:teuthology.orchestra.run.smithi160.stdout:34359738371 2020-11-19T02:51:42.627 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2020-11-19T02:51:43.628 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-19T02:51:43.999 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2020-11-19T02:51:44.017 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2020-11-19T02:51:44.018 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2020-11-19T02:51:44.300 INFO:teuthology.orchestra.run.smithi160.stdout:34359738371 2020-11-19T02:51:44.323 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2020-11-19T02:51:44.323 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2020-11-19T02:51:44.620 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2020-11-19T02:51:44.634 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2020-11-19T02:51:44.634 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-11-19T02:51:44.635 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-19T02:51:44.919 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2020-11-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-19T02:51:44.907108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.809754"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.104499+0000","last_change":"2020-11-19T02:51:39.101588+0000","last_active":"2020-11-19T02:51:39.104499+0000","last_peered":"2020-11-19T02:51:39.104499+0000","last_clean":"2020-11-19T02:51:39.104499+0000","last_became_active":"2020-11-19T02:51:39.101232+0000","last_became_peered":"2020-11-19T02:51:39.101232+0000","last_unstale":"2020-11-19T02:51:39.104499+0000","last_undegraded":"2020-11-19T02:51:39.104499+0000","last_fullsized":"2020-11-19T02:51:39.104499+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.102381+0000","last_change":"2020-11-19T02:51:39.101272+0000","last_active":"2020-11-19T02:51:39.102381+0000","last_peered":"2020-11-19T02:51:39.102381+0000","last_clean":"2020-11-19T02:51:39.102381+0000","last_became_active":"2020-11-19T02:51:39.100727+0000","last_became_peered":"2020-11-19T02:51:39.100727+0000","last_unstale":"2020-11-19T02:51:39.102381+0000","last_undegraded":"2020-11-19T02:51:39.102381+0000","last_fullsized":"2020-11-19T02:51:39.102381+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.104533+0000","last_change":"2020-11-19T02:51:39.101722+0000","last_active":"2020-11-19T02:51:39.104533+0000","last_peered":"2020-11-19T02:51:39.104533+0000","last_clean":"2020-11-19T02:51:39.104533+0000","last_became_active":"2020-11-19T02:51:39.101390+0000","last_became_peered":"2020-11-19T02:51:39.101390+0000","last_unstale":"2020-11-19T02:51:39.104533+0000","last_undegraded":"2020-11-19T02:51:39.104533+0000","last_fullsized":"2020-11-19T02:51:39.104533+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.105352+0000","last_change":"2020-11-19T02:51:39.104208+0000","last_active":"2020-11-19T02:51:39.105352+0000","last_peered":"2020-11-19T02:51:39.105352+0000","last_clean":"2020-11-19T02:51:39.105352+0000","last_became_active":"2020-11-19T02:51:39.103406+0000","last_became_peered":"2020-11-19T02:51:39.103406+0000","last_unstale":"2020-11-19T02:51:39.105352+0000","last_undegraded":"2020-11-19T02:51:39.105352+0000","last_fullsized":"2020-11-19T02:51:39.105352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.097102+0000","last_change":"2020-11-19T02:51:35.123129+0000","last_active":"2020-11-19T02:51:39.097102+0000","last_peered":"2020-11-19T02:51:39.097102+0000","last_clean":"2020-11-19T02:51:39.097102+0000","last_became_active":"2020-11-19T02:51:35.121700+0000","last_became_peered":"2020-11-19T02:51:35.121700+0000","last_unstale":"2020-11-19T02:51:39.097102+0000","last_undegraded":"2020-11-19T02:51:39.097102+0000","last_fullsized":"2020-11-19T02:51:39.097102+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:33.039404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:33.039404+0000","last_clean_scrub_stamp":"2020-11-19T02:51:33.039404+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.104506+0000","last_change":"2020-11-19T02:51:39.103318+0000","last_active":"2020-11-19T02:51:39.104506+0000","last_peered":"2020-11-19T02:51:39.104506+0000","last_clean":"2020-11-19T02:51:39.104506+0000","last_became_active":"2020-11-19T02:51:39.102833+0000","last_became_peered":"2020-11-19T02:51:39.102833+0000","last_unstale":"2020-11-19T02:51:39.104506+0000","last_undegraded":"2020-11-19T02:51:39.104506+0000","last_fullsized":"2020-11-19T02:51:39.104506+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.101660+0000","last_change":"2020-11-19T02:51:39.100770+0000","last_active":"2020-11-19T02:51:39.101660+0000","last_peered":"2020-11-19T02:51:39.101660+0000","last_clean":"2020-11-19T02:51:39.101660+0000","last_became_active":"2020-11-19T02:51:39.100266+0000","last_became_peered":"2020-11-19T02:51:39.100266+0000","last_unstale":"2020-11-19T02:51:39.101660+0000","last_undegraded":"2020-11-19T02:51:39.101660+0000","last_fullsized":"2020-11-19T02:51:39.101660+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.106001+0000","last_change":"2020-11-19T02:51:39.104325+0000","last_active":"2020-11-19T02:51:39.106001+0000","last_peered":"2020-11-19T02:51:39.106001+0000","last_clean":"2020-11-19T02:51:39.106001+0000","last_became_active":"2020-11-19T02:51:39.103970+0000","last_became_peered":"2020-11-19T02:51:39.103970+0000","last_unstale":"2020-11-19T02:51:39.106001+0000","last_undegraded":"2020-11-19T02:51:39.106001+0000","last_fullsized":"2020-11-19T02:51:39.106001+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.101705+0000","last_change":"2020-11-19T02:51:39.100240+0000","last_active":"2020-11-19T02:51:39.101705+0000","last_peered":"2020-11-19T02:51:39.101705+0000","last_clean":"2020-11-19T02:51:39.101705+0000","last_became_active":"2020-11-19T02:51:39.099899+0000","last_became_peered":"2020-11-19T02:51:39.099899+0000","last_unstale":"2020-11-19T02:51:39.101705+0000","last_undegraded":"2020-11-19T02:51:39.101705+0000","last_fullsized":"2020-11-19T02:51:39.101705+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.218}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]}]}]}} 2020-11-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-19T02:51:45.221 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:45.224 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2020-11-19T02:51:45.237 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-19T02:51:44.907108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.809754"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.104499+0000","last_change":"2020-11-19T02:51:39.101588+0000","last_active":"2020-11-19T02:51:39.104499+0000","last_peered":"2020-11-19T02:51:39.104499+0000","last_clean":"2020-11-19T02:51:39.104499+0000","last_became_active":"2020-11-19T02:51:39.101232+0000","last_became_peered":"2020-11-19T02:51:39.101232+0000","last_unstale":"2020-11-19T02:51:39.104499+0000","last_undegraded":"2020-11-19T02:51:39.104499+0000","last_fullsized":"2020-11-19T02:51:39.104499+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.102381+0000","last_change":"2020-11-19T02:51:39.101272+0000","last_active":"2020-11-19T02:51:39.102381+0000","last_peered":"2020-11-19T02:51:39.102381+0000","last_clean":"2020-11-19T02:51:39.102381+0000","last_became_active":"2020-11-19T02:51:39.100727+0000","last_became_peered":"2020-11-19T02:51:39.100727+0000","last_unstale":"2020-11-19T02:51:39.102381+0000","last_undegraded":"2020-11-19T02:51:39.102381+0000","last_fullsized":"2020-11-19T02:51:39.102381+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.104533+0000","last_change":"2020-11-19T02:51:39.101722+0000","last_active":"2020-11-19T02:51:39.104533+0000","last_peered":"2020-11-19T02:51:39.104533+0000","last_clean":"2020-11-19T02:51:39.104533+0000","last_became_active":"2020-11-19T02:51:39.101390+0000","last_became_peered":"2020-11-19T02:51:39.101390+0000","last_unstale":"2020-11-19T02:51:39.104533+0000","last_undegraded":"2020-11-19T02:51:39.104533+0000","last_fullsized":"2020-11-19T02:51:39.104533+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.105352+0000","last_change":"2020-11-19T02:51:39.104208+0000","last_active":"2020-11-19T02:51:39.105352+0000","last_peered":"2020-11-19T02:51:39.105352+0000","last_clean":"2020-11-19T02:51:39.105352+0000","last_became_active":"2020-11-19T02:51:39.103406+0000","last_became_peered":"2020-11-19T02:51:39.103406+0000","last_unstale":"2020-11-19T02:51:39.105352+0000","last_undegraded":"2020-11-19T02:51:39.105352+0000","last_fullsized":"2020-11-19T02:51:39.105352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.097102+0000","last_change":"2020-11-19T02:51:35.123129+0000","last_active":"2020-11-19T02:51:39.097102+0000","last_peered":"2020-11-19T02:51:39.097102+0000","last_clean":"2020-11-19T02:51:39.097102+0000","last_became_active":"2020-11-19T02:51:35.121700+0000","last_became_peered":"2020-11-19T02:51:35.121700+0000","last_unstale":"2020-11-19T02:51:39.097102+0000","last_undegraded":"2020-11-19T02:51:39.097102+0000","last_fullsized":"2020-11-19T02:51:39.097102+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:33.039404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:33.039404+0000","last_clean_scrub_stamp":"2020-11-19T02:51:33.039404+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.104506+0000","last_change":"2020-11-19T02:51:39.103318+0000","last_active":"2020-11-19T02:51:39.104506+0000","last_peered":"2020-11-19T02:51:39.104506+0000","last_clean":"2020-11-19T02:51:39.104506+0000","last_became_active":"2020-11-19T02:51:39.102833+0000","last_became_peered":"2020-11-19T02:51:39.102833+0000","last_unstale":"2020-11-19T02:51:39.104506+0000","last_undegraded":"2020-11-19T02:51:39.104506+0000","last_fullsized":"2020-11-19T02:51:39.104506+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.101660+0000","last_change":"2020-11-19T02:51:39.100770+0000","last_active":"2020-11-19T02:51:39.101660+0000","last_peered":"2020-11-19T02:51:39.101660+0000","last_clean":"2020-11-19T02:51:39.101660+0000","last_became_active":"2020-11-19T02:51:39.100266+0000","last_became_peered":"2020-11-19T02:51:39.100266+0000","last_unstale":"2020-11-19T02:51:39.101660+0000","last_undegraded":"2020-11-19T02:51:39.101660+0000","last_fullsized":"2020-11-19T02:51:39.101660+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.106001+0000","last_change":"2020-11-19T02:51:39.104325+0000","last_active":"2020-11-19T02:51:39.106001+0000","last_peered":"2020-11-19T02:51:39.106001+0000","last_clean":"2020-11-19T02:51:39.106001+0000","last_became_active":"2020-11-19T02:51:39.103970+0000","last_became_peered":"2020-11-19T02:51:39.103970+0000","last_unstale":"2020-11-19T02:51:39.106001+0000","last_undegraded":"2020-11-19T02:51:39.106001+0000","last_fullsized":"2020-11-19T02:51:39.106001+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T02:51:39.101705+0000","last_change":"2020-11-19T02:51:39.100240+0000","last_active":"2020-11-19T02:51:39.101705+0000","last_peered":"2020-11-19T02:51:39.101705+0000","last_clean":"2020-11-19T02:51:39.101705+0000","last_became_active":"2020-11-19T02:51:39.099899+0000","last_became_peered":"2020-11-19T02:51:39.099899+0000","last_unstale":"2020-11-19T02:51:39.101705+0000","last_undegraded":"2020-11-19T02:51:39.101705+0000","last_fullsized":"2020-11-19T02:51:39.101705+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T02:51:38.079084+0000","last_clean_scrub_stamp":"2020-11-19T02:51:38.079084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.218}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]}]}]}} 2020-11-19T02:51:45.241 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-11-19T02:51:45.243 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-11-19T02:51:45.244 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-11-19T02:51:45.246 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2020-11-19T02:51:45.590 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T02:51:45.607 INFO:teuthology.orchestra.run.smithi160.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-19T02:51:45.609 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-11-19T02:51:45.610 INFO:teuthology.run_tasks:Running task rgw... 2020-11-19T02:51:45.642 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-11-19T02:51:45.643 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-11-19T02:51:45.644 INFO:tasks.rgw:Creating data pools 2020-11-19T02:51:45.645 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-11-19T02:51:45.646 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-19T02:51:46.032 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'default.rgw.buckets.data' created 2020-11-19T02:51:47.042 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-19T02:51:48.031 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-19T02:51:48.044 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-19T02:51:49.047 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'default.rgw.buckets.index' created 2020-11-19T02:51:49.061 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-19T02:51:50.070 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-19T02:51:50.093 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-11-19T02:51:50.093 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-19T02:51:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-19T02:51:50.387 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-19T02:51:51.054 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-19T02:51:51.270 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-19T02:51:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-19T02:51:51.586 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-19T02:51:52.068 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-19T02:51:52.082 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-11-19T02:51:52.083 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-19T02:51:53.163 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.data' created 2020-11-19T02:51:53.181 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-19T02:51:54.183 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-19T02:51:54.208 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-19T02:51:55.183 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.index' created 2020-11-19T02:51:55.194 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-19T02:51:56.197 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-19T02:51:56.218 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-11-19T02:51:56.219 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-19T02:51:56.504 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-19T02:51:56.515 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-19T02:51:57.188 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-19T02:51:57.505 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-19T02:51:57.516 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-19T02:51:58.193 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-19T02:51:58.205 DEBUG:tasks.rgw:Pools created 2020-11-19T02:51:58.205 INFO:tasks.rgw:Configuring compression type = random 2020-11-19T02:51:58.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-11-19T02:51:58.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-11-19T02:51:58.207 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-11-19T02:51:58.243 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:51:58.243 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:51:58.252 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:51:58.247+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.109 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.103+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.109 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.103+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.112 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.107+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.112 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.107+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.113 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.111+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.113 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.111+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.113 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.111+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.131 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.132 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.132 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.132 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.133 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.133 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.133 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-19T02:52:00.133 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.134 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.134 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.127+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.135 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.131+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.135 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.131+0000 7fec9940f0c0 10 Creating default zonegroup 2020-11-19T02:52:00.135 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.131+0000 7fec9940f0c0 20 get_system_obj_state: rctx=0x7fff1a551b80 obj=:.pools.avail state=0x55e6af72d2e0 s->prefetch_data=0 2020-11-19T02:52:00.135 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.131+0000 7fec9940f0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-19T02:52:00.152 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.147+0000 7fec9940f0c0 10 WARNING: store->list_zones() returned r=-2 2020-11-19T02:52:00.152 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.147+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.153 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.151+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.158 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.155+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.159 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.155+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.159 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.155+0000 7fec9940f0c0 10 could not read realm id: (2) No such file or directory 2020-11-19T02:52:00.160 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.155+0000 7fec9940f0c0 10 WARNING: failed to set zone as default, r=-22 2020-11-19T02:52:00.160 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.155+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.160 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.155+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.166 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.163+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.167 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.163+0000 7fec9940f0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T02:52:00.167 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.163+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.168 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.163+0000 7fec9940f0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:00.168 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.163+0000 7fec9940f0c0 20 zonegroup default 2020-11-19T02:52:00.169 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.163+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.169 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.167+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:00.169 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.167+0000 7fec9940f0c0 10 Cannot find current period zone using local zone 2020-11-19T02:52:00.170 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.167+0000 7fec9940f0c0 10 Using default name default 2020-11-19T02:52:00.170 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.167+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.170 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.167+0000 7fec9940f0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:00.171 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.167+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:00.171 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.167+0000 7fec9940f0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T02:52:00.171 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.167+0000 7fec9940f0c0 20 zone default found 2020-11-19T02:52:00.172 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:00.167+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:02.133 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:02.131+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:02.134 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:02.131+0000 7fec9940f0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:02.134 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:02.131+0000 7fec9940f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:02.134 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:02.131+0000 7fec9940f0c0 20 started sync module instance, tier type = 2020-11-19T02:52:02.135 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:02.131+0000 7fec9940f0c0 20 started zone id=2cd49086-2a16-48c3-a1a7-0b4ff6f92950 (name=default) with tier type = 2020-11-19T02:52:04.195 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.191+0000 7fec9940f0c0 20 add_watcher() i=0 2020-11-19T02:52:04.196 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.191+0000 7fec9940f0c0 20 add_watcher() i=1 2020-11-19T02:52:04.196 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.191+0000 7fec9940f0c0 20 add_watcher() i=2 2020-11-19T02:52:04.196 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.191+0000 7fec9940f0c0 20 add_watcher() i=3 2020-11-19T02:52:04.196 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.191+0000 7fec9940f0c0 20 add_watcher() i=4 2020-11-19T02:52:04.197 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.191+0000 7fec9940f0c0 20 add_watcher() i=5 2020-11-19T02:52:04.197 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.191+0000 7fec9940f0c0 20 add_watcher() i=6 2020-11-19T02:52:04.197 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.195+0000 7fec9940f0c0 20 add_watcher() i=7 2020-11-19T02:52:04.198 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.195+0000 7fec9940f0c0 2 all 8 watchers are set, enabling cache 2020-11-19T02:52:04.199 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.195+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:04.200 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.195+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:04.200 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.199+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:04.201 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.199+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:04.201 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.199+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.199+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:04.208 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.203+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:04.213 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.211+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:04.215 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.211+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:04.218 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.215+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:04.221 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.219+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:04.225 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.223+0000 7fec9940f0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:04.229 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.223+0000 7fec567e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T02:52:04.229 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.227+0000 7fec9940f0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T02:52:04.229 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:04.227+0000 7fec9940f0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T02:52:08.133 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.127+0000 7fec9940f0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-19T02:52:08.274 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.271+0000 7febc57fa700 20 reqs_thread_entry: start 2020-11-19T02:52:08.274 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.271+0000 7fec9940f0c0 20 init_complete bucket index max shards: 11 2020-11-19T02:52:08.275 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.271+0000 7febb2ffd700 20 reqs_thread_entry: start 2020-11-19T02:52:08.275 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.271+0000 7fec9940f0c0 10 Started notification manager with: 1 workers 2020-11-19T02:52:08.275 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.271+0000 7febb0ff9700 20 reqs_thread_entry: start 2020-11-19T02:52:08.276 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.271+0000 7febb17fa700 20 INFO: next queues processing will happen at: Thu Nov 19 02:52:38 2020 2020-11-19T02:52:08.276 INFO:teuthology.orchestra.run.smithi086.stderr: 2020-11-19T02:52:08.299 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T02:52:08.299 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.295+0000 7fec9940f0c0 20 remove_watcher() i=0 2020-11-19T02:52:08.299 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.295+0000 7fec9940f0c0 2 removed watcher, disabling cache 2020-11-19T02:52:08.304 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.299+0000 7fec9940f0c0 20 remove_watcher() i=1 2020-11-19T02:52:08.306 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.303+0000 7fec9940f0c0 20 remove_watcher() i=2 2020-11-19T02:52:08.309 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.307+0000 7fec9940f0c0 20 remove_watcher() i=3 2020-11-19T02:52:08.315 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.311+0000 7fec9940f0c0 20 remove_watcher() i=4 2020-11-19T02:52:08.318 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.315+0000 7fec9940f0c0 20 remove_watcher() i=5 2020-11-19T02:52:08.320 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.315+0000 7fec9940f0c0 20 remove_watcher() i=6 2020-11-19T02:52:08.323 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.319+0000 7fec9940f0c0 20 remove_watcher() i=7 2020-11-19T02:52:08.325 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.323+0000 7fec567e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T02:52:08.333 DEBUG:tasks.util.rgw: json result: [] 2020-11-19T02:52:08.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T02:52:08.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T02:52:08.334 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-19T02:52:08.420 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:08.421 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:08.429 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.427+0000 7fe1591640c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T02:52:08.430 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.427+0000 7fe1591640c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T02:52:08.435 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.431+0000 7fe08e7fc700 20 reqs_thread_entry: start 2020-11-19T02:52:08.435 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.431+0000 7fe1591640c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.436 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.431+0000 7fe1591640c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:08.436 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.431+0000 7fe1591640c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.438 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.435+0000 7fe1591640c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T02:52:08.438 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.435+0000 7fe1591640c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.440 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.435+0000 7fe1591640c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.440 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.435+0000 7fe1591640c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.441 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.435+0000 7fe1591640c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:08.441 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.435+0000 7fe1591640c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.442 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.439+0000 7fe1591640c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:08.451 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:52:08.452 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "2cd49086-2a16-48c3-a1a7-0b4ff6f92950", 2020-11-19T02:52:08.452 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default", 2020-11-19T02:52:08.452 INFO:teuthology.orchestra.run.smithi086.stdout: "domain_root": "default.rgw.meta:root", 2020-11-19T02:52:08.453 INFO:teuthology.orchestra.run.smithi086.stdout: "control_pool": "default.rgw.control", 2020-11-19T02:52:08.453 INFO:teuthology.orchestra.run.smithi086.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-19T02:52:08.454 INFO:teuthology.orchestra.run.smithi086.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-19T02:52:08.454 INFO:teuthology.orchestra.run.smithi086.stdout: "log_pool": "default.rgw.log", 2020-11-19T02:52:08.455 INFO:teuthology.orchestra.run.smithi086.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-19T02:52:08.455 INFO:teuthology.orchestra.run.smithi086.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-19T02:52:08.455 INFO:teuthology.orchestra.run.smithi086.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-19T02:52:08.456 INFO:teuthology.orchestra.run.smithi086.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-19T02:52:08.456 INFO:teuthology.orchestra.run.smithi086.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-19T02:52:08.456 INFO:teuthology.orchestra.run.smithi086.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-19T02:52:08.457 INFO:teuthology.orchestra.run.smithi086.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-19T02:52:08.457 INFO:teuthology.orchestra.run.smithi086.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-19T02:52:08.458 INFO:teuthology.orchestra.run.smithi086.stdout: "otp_pool": "default.rgw.otp", 2020-11-19T02:52:08.458 INFO:teuthology.orchestra.run.smithi086.stdout: "system_key": { 2020-11-19T02:52:08.458 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "", 2020-11-19T02:52:08.459 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "" 2020-11-19T02:52:08.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:08.459 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_pools": [ 2020-11-19T02:52:08.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:08.460 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "default-placement", 2020-11-19T02:52:08.461 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:52:08.461 INFO:teuthology.orchestra.run.smithi086.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-19T02:52:08.461 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": { 2020-11-19T02:52:08.462 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD": { 2020-11-19T02:52:08.463 INFO:teuthology.orchestra.run.smithi086.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-19T02:52:08.463 INFO:teuthology.orchestra.run.smithi086.stdout: "compression_type": "random" 2020-11-19T02:52:08.464 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:08.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:08.465 INFO:teuthology.orchestra.run.smithi086.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-19T02:52:08.465 INFO:teuthology.orchestra.run.smithi086.stdout: "index_type": 0 2020-11-19T02:52:08.465 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:08.466 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:08.466 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:08.466 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "", 2020-11-19T02:52:08.467 INFO:teuthology.orchestra.run.smithi086.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-19T02:52:08.467 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:52:08.469 DEBUG:tasks.util.rgw: json result: {'id': '2cd49086-2a16-48c3-a1a7-0b4ff6f92950', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-19T02:52:08.469 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-11-19T02:52:08.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-11-19T02:52:08.475 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-11-19T02:52:08.548 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:08.549 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:08.562 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.559+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.564 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.559+0000 7fed1422a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.564 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.559+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.565 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.563+0000 7fed1422a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.565 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.563+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.565 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.563+0000 7fed1422a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.566 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.563+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.566 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.563+0000 7fed1422a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:08.589 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 RGWRados::pool_iterate: got zonegroup_info.777cdd47-35ae-4752-a84d-f783b45f0b26 2020-11-19T02:52:08.589 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-19T02:52:08.589 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-19T02:52:08.590 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 RGWRados::pool_iterate: got zone_info.2cd49086-2a16-48c3-a1a7-0b4ff6f92950 2020-11-19T02:52:08.590 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.591 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:08.591 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.591 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T02:52:08.591 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.592 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:08.592 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.592 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:08.593 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.587+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.593 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.593 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-19T02:52:08.594 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.594 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.594 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.595 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:08.595 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.595 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:08.596 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 zonegroup default 2020-11-19T02:52:08.596 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.596 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.597 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 10 Cannot find current period zone using local zone 2020-11-19T02:52:08.597 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.591+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.597 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.598 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.598 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:08.599 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.599 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T02:52:08.599 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 zone default found 2020-11-19T02:52:08.599 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.600 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.600 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.600 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.601 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 started sync module instance, tier type = 2020-11-19T02:52:08.601 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.595+0000 7fed1422a0c0 20 started zone id=2cd49086-2a16-48c3-a1a7-0b4ff6f92950 (name=default) with tier type = 2020-11-19T02:52:08.626 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.623+0000 7fed1422a0c0 20 add_watcher() i=0 2020-11-19T02:52:08.627 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.623+0000 7fed1422a0c0 20 add_watcher() i=1 2020-11-19T02:52:08.627 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.623+0000 7fed1422a0c0 20 add_watcher() i=2 2020-11-19T02:52:08.627 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.623+0000 7fed1422a0c0 20 add_watcher() i=3 2020-11-19T02:52:08.627 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.623+0000 7fed1422a0c0 20 add_watcher() i=4 2020-11-19T02:52:08.627 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.623+0000 7fed1422a0c0 20 add_watcher() i=5 2020-11-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.623+0000 7fed1422a0c0 20 add_watcher() i=6 2020-11-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.623+0000 7fed1422a0c0 20 add_watcher() i=7 2020-11-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.623+0000 7fed1422a0c0 2 all 8 watchers are set, enabling cache 2020-11-19T02:52:08.631 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.627+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.627+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.627+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T02:52:08.633 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.627+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T02:52:08.634 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.627+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T02:52:08.634 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.627+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T02:52:08.634 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.627+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:08.635 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T02:52:08.635 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T02:52:08.637 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T02:52:08.637 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T02:52:08.637 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T02:52:08.638 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:08.638 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T02:52:08.638 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T02:52:08.638 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T02:52:08.639 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T02:52:08.639 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T02:52:08.639 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.631+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:08.640 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.635+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T02:52:08.640 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.635+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T02:52:08.640 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.635+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T02:52:08.640 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.635+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T02:52:08.641 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.635+0000 7fed1422a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T02:52:08.641 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.635+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:08.641 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.639+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:08.642 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.639+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:08.644 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.639+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:08.645 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.643+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:08.655 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.643+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:08.655 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.643+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:08.656 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.647+0000 7fed1422a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:08.656 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.647+0000 7fecd17e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T02:52:08.657 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.647+0000 7fed1422a0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T02:52:08.657 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.647+0000 7fed1422a0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T02:52:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.743+0000 7fec3ffff700 20 reqs_thread_entry: start 2020-11-19T02:52:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.743+0000 7fec3dffb700 20 reqs_thread_entry: start 2020-11-19T02:52:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.743+0000 7fed1422a0c0 20 init_complete bucket index max shards: 11 2020-11-19T02:52:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.743+0000 7fed1422a0c0 10 Started notification manager with: 1 workers 2020-11-19T02:52:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.743+0000 7fec2bfff700 20 INFO: next queues processing will happen at: Thu Nov 19 02:52:39 2020 2020-11-19T02:52:08.747 INFO:teuthology.orchestra.run.smithi086.stderr: 2020-11-19T02:52:08.748 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.743+0000 7fec2b7fe700 20 reqs_thread_entry: start 2020-11-19T02:52:08.776 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T02:52:08.777 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.771+0000 7fed1422a0c0 20 remove_watcher() i=0 2020-11-19T02:52:08.778 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.771+0000 7fed1422a0c0 2 removed watcher, disabling cache 2020-11-19T02:52:08.779 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.775+0000 7fed1422a0c0 20 remove_watcher() i=1 2020-11-19T02:52:08.781 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.779+0000 7fed1422a0c0 20 remove_watcher() i=2 2020-11-19T02:52:08.784 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.779+0000 7fed1422a0c0 20 remove_watcher() i=3 2020-11-19T02:52:08.787 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.783+0000 7fed1422a0c0 20 remove_watcher() i=4 2020-11-19T02:52:08.790 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.787+0000 7fed1422a0c0 20 remove_watcher() i=5 2020-11-19T02:52:08.796 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.791+0000 7fed1422a0c0 20 remove_watcher() i=6 2020-11-19T02:52:08.798 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.795+0000 7fed1422a0c0 20 remove_watcher() i=7 2020-11-19T02:52:08.800 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.795+0000 7fecd17e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T02:52:08.807 DEBUG:tasks.util.rgw: json result: [] 2020-11-19T02:52:08.808 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T02:52:08.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T02:52:08.808 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-19T02:52:08.887 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:08.888 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:08.899 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.895+0000 7f9bc0af00c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T02:52:08.899 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.895+0000 7f9bc0af00c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T02:52:08.906 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.903+0000 7f9bc0af00c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.907 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.903+0000 7f9af5ffb700 20 reqs_thread_entry: start 2020-11-19T02:52:08.907 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.903+0000 7f9bc0af00c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:08.908 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.903+0000 7f9bc0af00c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.909 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.907+0000 7f9bc0af00c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T02:52:08.909 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.907+0000 7f9bc0af00c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.910 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.907+0000 7f9bc0af00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:08.910 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.907+0000 7f9bc0af00c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.911 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.907+0000 7f9bc0af00c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:08.911 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.907+0000 7f9bc0af00c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:08.911 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:08.907+0000 7f9bc0af00c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:08.918 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:52:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "2cd49086-2a16-48c3-a1a7-0b4ff6f92950", 2020-11-19T02:52:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default", 2020-11-19T02:52:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "domain_root": "default.rgw.meta:root", 2020-11-19T02:52:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "control_pool": "default.rgw.control", 2020-11-19T02:52:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-19T02:52:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-19T02:52:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "log_pool": "default.rgw.log", 2020-11-19T02:52:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-19T02:52:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-19T02:52:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-19T02:52:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-19T02:52:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-19T02:52:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-19T02:52:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-19T02:52:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-19T02:52:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "otp_pool": "default.rgw.otp", 2020-11-19T02:52:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "system_key": { 2020-11-19T02:52:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "", 2020-11-19T02:52:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "" 2020-11-19T02:52:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_pools": [ 2020-11-19T02:52:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "default-placement", 2020-11-19T02:52:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:52:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-19T02:52:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": { 2020-11-19T02:52:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD": { 2020-11-19T02:52:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-19T02:52:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "compression_type": "random" 2020-11-19T02:52:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-19T02:52:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "index_type": 0 2020-11-19T02:52:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "", 2020-11-19T02:52:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-19T02:52:08.926 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:52:08.927 DEBUG:tasks.util.rgw: json result: {'id': '2cd49086-2a16-48c3-a1a7-0b4ff6f92950', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-19T02:52:08.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-11-19T02:52:08.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-11-19T02:52:08.929 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-11-19T02:52:08.965 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:08.965 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:08.974 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:08.969+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.226 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.221+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.227 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.221+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.233 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.229+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.235 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.229+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.237 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.229+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.237 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.229+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.233+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.256 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.256 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.257 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.258 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.258 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.259 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.259 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 10 cannot find current period zonegroup using local zonegroup 2020-11-19T02:52:10.259 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.259 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.259 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.260 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.260 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 10 Creating default zonegroup 2020-11-19T02:52:10.260 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 20 get_system_obj_state: rctx=0x7fffe8255520 obj=:.pools.avail state=0x5624ec5bf2e0 s->prefetch_data=0 2020-11-19T02:52:10.260 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.253+0000 7f5b0a7010c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-19T02:52:10.272 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.269+0000 7f5b0a7010c0 10 WARNING: store->list_zones() returned r=-2 2020-11-19T02:52:10.273 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.269+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.273 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.269+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.278 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.273+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.278 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.277+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.279 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.277+0000 7f5b0a7010c0 10 could not read realm id: (2) No such file or directory 2020-11-19T02:52:10.279 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.277+0000 7f5b0a7010c0 10 WARNING: failed to set zone as default, r=-22 2020-11-19T02:52:10.279 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.277+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.277+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.288 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.285+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.289 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.285+0000 7f5b0a7010c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T02:52:10.289 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.285+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.290 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.285+0000 7f5b0a7010c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:10.290 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.285+0000 7f5b0a7010c0 20 zonegroup default 2020-11-19T02:52:10.290 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.285+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.292 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.289+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:10.292 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.289+0000 7f5b0a7010c0 10 Cannot find current period zone using local zone 2020-11-19T02:52:10.293 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.289+0000 7f5b0a7010c0 10 Using default name default 2020-11-19T02:52:10.293 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.289+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.293 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.289+0000 7f5b0a7010c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:10.293 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.289+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:10.293 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.289+0000 7f5b0a7010c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T02:52:10.294 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.289+0000 7f5b0a7010c0 20 zone default found 2020-11-19T02:52:10.294 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:10.289+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:12.248 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:12.245+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:12.248 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:12.245+0000 7f5b0a7010c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:12.249 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:12.245+0000 7f5b0a7010c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:12.249 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:12.245+0000 7f5b0a7010c0 20 started sync module instance, tier type = 2020-11-19T02:52:12.249 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:12.245+0000 7f5b0a7010c0 20 started zone id=3f662eea-1344-4da5-823e-bc1af50af572 (name=default) with tier type = 2020-11-19T02:52:14.306 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.301+0000 7f5b0a7010c0 20 add_watcher() i=0 2020-11-19T02:52:14.307 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.301+0000 7f5b0a7010c0 20 add_watcher() i=1 2020-11-19T02:52:14.309 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.301+0000 7f5b0a7010c0 20 add_watcher() i=2 2020-11-19T02:52:14.310 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.301+0000 7f5b0a7010c0 20 add_watcher() i=3 2020-11-19T02:52:14.311 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.301+0000 7f5b0a7010c0 20 add_watcher() i=4 2020-11-19T02:52:14.312 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.301+0000 7f5b0a7010c0 20 add_watcher() i=5 2020-11-19T02:52:14.312 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.301+0000 7f5b0a7010c0 20 add_watcher() i=6 2020-11-19T02:52:14.316 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.305+0000 7f5b0a7010c0 20 add_watcher() i=7 2020-11-19T02:52:14.316 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.305+0000 7f5b0a7010c0 2 all 8 watchers are set, enabling cache 2020-11-19T02:52:14.316 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.305+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:14.317 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.305+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:14.317 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.305+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:14.317 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.309+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:14.317 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.309+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:14.317 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.309+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:14.318 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.313+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:14.320 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.317+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:14.324 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.321+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:14.327 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.325+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:14.331 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.329+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:14.335 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.333+0000 7f5b0a7010c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:14.336 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.333+0000 7f5ac7fdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T02:52:14.336 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.333+0000 7f5b0a7010c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T02:52:14.336 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:14.333+0000 7f5b0a7010c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T02:52:18.268 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.265+0000 7f5b0a7010c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-19T02:52:18.408 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.405+0000 7f5a2effd700 20 reqs_thread_entry: start 2020-11-19T02:52:18.409 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.405+0000 7f5a2cff9700 20 reqs_thread_entry: start 2020-11-19T02:52:18.409 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.405+0000 7f5b0a7010c0 20 init_complete bucket index max shards: 11 2020-11-19T02:52:18.409 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.405+0000 7f5b0a7010c0 10 Started notification manager with: 1 workers 2020-11-19T02:52:18.410 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.405+0000 7f5a1effd700 20 INFO: next queues processing will happen at: Thu Nov 19 02:52:48 2020 2020-11-19T02:52:18.410 INFO:teuthology.orchestra.run.smithi160.stderr: 2020-11-19T02:52:18.410 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.405+0000 7f5a1e7fc700 20 reqs_thread_entry: start 2020-11-19T02:52:18.440 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2020-11-19T02:52:18.441 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.437+0000 7f5b0a7010c0 20 remove_watcher() i=0 2020-11-19T02:52:18.441 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.437+0000 7f5b0a7010c0 2 removed watcher, disabling cache 2020-11-19T02:52:18.443 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.441+0000 7f5b0a7010c0 20 remove_watcher() i=1 2020-11-19T02:52:18.446 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.441+0000 7f5b0a7010c0 20 remove_watcher() i=2 2020-11-19T02:52:18.448 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.445+0000 7f5b0a7010c0 20 remove_watcher() i=3 2020-11-19T02:52:18.452 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.449+0000 7f5b0a7010c0 20 remove_watcher() i=4 2020-11-19T02:52:18.454 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.453+0000 7f5b0a7010c0 20 remove_watcher() i=5 2020-11-19T02:52:18.457 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.453+0000 7f5b0a7010c0 20 remove_watcher() i=6 2020-11-19T02:52:18.460 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.457+0000 7f5b0a7010c0 20 remove_watcher() i=7 2020-11-19T02:52:18.461 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.457+0000 7f5ac7fdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T02:52:18.478 DEBUG:tasks.util.rgw: json result: [] 2020-11-19T02:52:18.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T02:52:18.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T02:52:18.479 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-19T02:52:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:18.583 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.581+0000 7fb3ce9840c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T02:52:18.584 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.581+0000 7fb3ce9840c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T02:52:18.589 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.585+0000 7fb303fff700 20 reqs_thread_entry: start 2020-11-19T02:52:18.589 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.585+0000 7fb3ce9840c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.591 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.589+0000 7fb3ce9840c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:18.591 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.589+0000 7fb3ce9840c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.593 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.589+0000 7fb3ce9840c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T02:52:18.593 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.589+0000 7fb3ce9840c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.594 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.589+0000 7fb3ce9840c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.594 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.589+0000 7fb3ce9840c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.594 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.593+0000 7fb3ce9840c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:18.595 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.593+0000 7fb3ce9840c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.595 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.593+0000 7fb3ce9840c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:18.607 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:52:18.608 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "3f662eea-1344-4da5-823e-bc1af50af572", 2020-11-19T02:52:18.608 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default", 2020-11-19T02:52:18.609 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "default.rgw.meta:root", 2020-11-19T02:52:18.610 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "default.rgw.control", 2020-11-19T02:52:18.610 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-19T02:52:18.611 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-19T02:52:18.611 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "default.rgw.log", 2020-11-19T02:52:18.612 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-19T02:52:18.613 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-19T02:52:18.613 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-19T02:52:18.616 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-19T02:52:18.617 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-19T02:52:18.618 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-19T02:52:18.618 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-19T02:52:18.619 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-19T02:52:18.620 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "default.rgw.otp", 2020-11-19T02:52:18.622 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T02:52:18.625 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "", 2020-11-19T02:52:18.626 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "" 2020-11-19T02:52:18.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:18.629 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T02:52:18.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:18.631 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T02:52:18.632 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:52:18.635 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-19T02:52:18.635 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T02:52:18.636 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T02:52:18.637 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-19T02:52:18.638 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2020-11-19T02:52:18.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:18.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:18.639 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-19T02:52:18.639 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T02:52:18.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:18.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:18.639 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:18.640 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "", 2020-11-19T02:52:18.640 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-19T02:52:18.640 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:52:18.641 DEBUG:tasks.util.rgw: json result: {'id': '3f662eea-1344-4da5-823e-bc1af50af572', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-19T02:52:18.642 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-11-19T02:52:18.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-11-19T02:52:18.643 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-11-19T02:52:18.725 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:18.725 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:18.735 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.733+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.738 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.737+0000 7f9327fdf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.739 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.737+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.740 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.737+0000 7f9327fdf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.741 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.737+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.741 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.737+0000 7f9327fdf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.742 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.737+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.743 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.737+0000 7f9327fdf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:18.768 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.765+0000 7f9327fdf0c0 20 RGWRados::pool_iterate: got zone_info.3f662eea-1344-4da5-823e-bc1af50af572 2020-11-19T02:52:18.769 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.765+0000 7f9327fdf0c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-19T02:52:18.769 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.765+0000 7f9327fdf0c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-19T02:52:18.769 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.765+0000 7f9327fdf0c0 20 RGWRados::pool_iterate: got zonegroup_info.529d7448-a860-4deb-bfa6-a8024ca4b61f 2020-11-19T02:52:18.769 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.765+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.770 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.765+0000 7f9327fdf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:18.770 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.765+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.771 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T02:52:18.771 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.771 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:18.772 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.772 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:18.773 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.773 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.773 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-19T02:52:18.774 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.774 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.774 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.769+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.775 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:18.775 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.776 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 zonegroup default 2020-11-19T02:52:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 10 Cannot find current period zone using local zone 2020-11-19T02:52:18.777 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.778 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.778 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.773+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.777+0000 7f9327fdf0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T02:52:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.777+0000 7f9327fdf0c0 20 zone default found 2020-11-19T02:52:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.777+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.780 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.777+0000 7f9327fdf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.781 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.777+0000 7f9327fdf0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:18.781 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.777+0000 7f9327fdf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:18.781 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.777+0000 7f9327fdf0c0 20 started sync module instance, tier type = 2020-11-19T02:52:18.782 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.777+0000 7f9327fdf0c0 20 started zone id=3f662eea-1344-4da5-823e-bc1af50af572 (name=default) with tier type = 2020-11-19T02:52:18.812 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.809+0000 7f9327fdf0c0 20 add_watcher() i=0 2020-11-19T02:52:18.813 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.809+0000 7f9327fdf0c0 20 add_watcher() i=1 2020-11-19T02:52:18.813 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.809+0000 7f9327fdf0c0 20 add_watcher() i=2 2020-11-19T02:52:18.813 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.809+0000 7f9327fdf0c0 20 add_watcher() i=3 2020-11-19T02:52:18.813 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.809+0000 7f9327fdf0c0 20 add_watcher() i=4 2020-11-19T02:52:18.813 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.809+0000 7f9327fdf0c0 20 add_watcher() i=5 2020-11-19T02:52:18.814 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.809+0000 7f9327fdf0c0 20 add_watcher() i=6 2020-11-19T02:52:18.814 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.813+0000 7f9327fdf0c0 20 add_watcher() i=7 2020-11-19T02:52:18.814 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.813+0000 7f9327fdf0c0 2 all 8 watchers are set, enabling cache 2020-11-19T02:52:18.816 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.813+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:18.817 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.813+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T02:52:18.817 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.813+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T02:52:18.818 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.813+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T02:52:18.818 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.813+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T02:52:18.818 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T02:52:18.819 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:18.819 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T02:52:18.820 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T02:52:18.820 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T02:52:18.820 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T02:52:18.821 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T02:52:18.822 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:18.822 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T02:52:18.823 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T02:52:18.823 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T02:52:18.823 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.817+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T02:52:18.825 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.821+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T02:52:18.825 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.821+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:18.826 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.821+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T02:52:18.826 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.821+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T02:52:18.826 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.821+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T02:52:18.827 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.821+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T02:52:18.827 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.821+0000 7f9327fdf0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T02:52:18.827 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.821+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:18.828 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.825+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:18.829 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.825+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:18.830 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.825+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:18.830 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.825+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:18.832 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.829+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:18.833 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.829+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T02:52:18.835 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.833+0000 7f9327fdf0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T02:52:18.836 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.833+0000 7f92e57e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T02:52:18.836 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.833+0000 7f9327fdf0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T02:52:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.833+0000 7f9327fdf0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T02:52:18.931 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.925+0000 7f9253fff700 20 reqs_thread_entry: start 2020-11-19T02:52:18.932 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.929+0000 7f9251ffb700 20 reqs_thread_entry: start 2020-11-19T02:52:18.932 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.929+0000 7f9327fdf0c0 20 init_complete bucket index max shards: 11 2020-11-19T02:52:18.932 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.929+0000 7f9327fdf0c0 10 Started notification manager with: 1 workers 2020-11-19T02:52:18.932 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.929+0000 7f923ffff700 20 INFO: next queues processing will happen at: Thu Nov 19 02:52:49 2020 2020-11-19T02:52:18.932 INFO:teuthology.orchestra.run.smithi160.stderr: 2020-11-19T02:52:18.933 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.929+0000 7f923f7fe700 20 reqs_thread_entry: start 2020-11-19T02:52:18.961 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2020-11-19T02:52:18.961 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.957+0000 7f9327fdf0c0 20 remove_watcher() i=0 2020-11-19T02:52:18.962 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.957+0000 7f9327fdf0c0 2 removed watcher, disabling cache 2020-11-19T02:52:18.964 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.961+0000 7f9327fdf0c0 20 remove_watcher() i=1 2020-11-19T02:52:18.966 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.965+0000 7f9327fdf0c0 20 remove_watcher() i=2 2020-11-19T02:52:18.971 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.969+0000 7f9327fdf0c0 20 remove_watcher() i=3 2020-11-19T02:52:18.974 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.973+0000 7f9327fdf0c0 20 remove_watcher() i=4 2020-11-19T02:52:18.977 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.973+0000 7f9327fdf0c0 20 remove_watcher() i=5 2020-11-19T02:52:18.980 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.977+0000 7f9327fdf0c0 20 remove_watcher() i=6 2020-11-19T02:52:18.983 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.981+0000 7f9327fdf0c0 20 remove_watcher() i=7 2020-11-19T02:52:18.984 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:18.981+0000 7f92e57e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T02:52:18.991 DEBUG:tasks.util.rgw: json result: [] 2020-11-19T02:52:18.992 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T02:52:18.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T02:52:18.993 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-19T02:52:19.075 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:19.075 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:19.087 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.085+0000 7f56a610d0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T02:52:19.088 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.085+0000 7f56a610d0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T02:52:19.095 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.093+0000 7f55da7fc700 20 reqs_thread_entry: start 2020-11-19T02:52:19.096 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.093+0000 7f56a610d0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:19.097 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.093+0000 7f56a610d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:19.097 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.093+0000 7f56a610d0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:19.099 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.097+0000 7f56a610d0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T02:52:19.100 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.097+0000 7f56a610d0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:19.100 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.097+0000 7f56a610d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T02:52:19.100 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.097+0000 7f56a610d0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:19.101 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.097+0000 7f56a610d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T02:52:19.101 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.097+0000 7f56a610d0c0 20 rados->read ofs=0 len=0 2020-11-19T02:52:19.101 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:19.097+0000 7f56a610d0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T02:52:19.107 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:52:19.108 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "3f662eea-1344-4da5-823e-bc1af50af572", 2020-11-19T02:52:19.108 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default", 2020-11-19T02:52:19.108 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "default.rgw.meta:root", 2020-11-19T02:52:19.109 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "default.rgw.control", 2020-11-19T02:52:19.109 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-19T02:52:19.109 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-19T02:52:19.109 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "default.rgw.log", 2020-11-19T02:52:19.109 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-19T02:52:19.110 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-19T02:52:19.110 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-19T02:52:19.110 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-19T02:52:19.110 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-19T02:52:19.111 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-19T02:52:19.111 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-19T02:52:19.111 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-19T02:52:19.111 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "default.rgw.otp", 2020-11-19T02:52:19.111 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T02:52:19.112 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "", 2020-11-19T02:52:19.112 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "" 2020-11-19T02:52:19.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:19.112 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T02:52:19.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:19.113 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T02:52:19.113 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:52:19.113 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-19T02:52:19.115 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T02:52:19.115 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T02:52:19.115 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-19T02:52:19.116 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2020-11-19T02:52:19.116 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:19.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:19.116 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-19T02:52:19.117 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T02:52:19.117 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:19.117 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:19.118 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:19.118 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "", 2020-11-19T02:52:19.118 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-19T02:52:19.118 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:52:19.119 DEBUG:tasks.util.rgw: json result: {'id': '3f662eea-1344-4da5-823e-bc1af50af572', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-19T02:52:19.120 INFO:tasks.rgw:Starting rgw... 2020-11-19T02:52:19.120 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-11-19T02:52:19.121 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-19T02:52:19.121 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-19T02:52:19.122 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-19T02:52:19.126 INFO:tasks.rgw.c1.client.0:Started 2020-11-19T02:52:19.126 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-11-19T02:52:19.127 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-19T02:52:19.128 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-19T02:52:19.128 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-19T02:52:19.133 INFO:tasks.rgw.c1.client.1:Started 2020-11-19T02:52:19.133 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-11-19T02:52:19.133 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-19T02:52:19.134 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-19T02:52:19.134 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-19T02:52:19.167 INFO:tasks.rgw.c2.client.0:Started 2020-11-19T02:52:19.168 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-11-19T02:52:19.168 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-19T02:52:19.169 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-19T02:52:19.169 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-19T02:52:19.184 INFO:tasks.rgw.c2.client.1:Started 2020-11-19T02:52:19.185 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi086.front.sepia.ceph.com:8000/ 2020-11-19T02:52:19.185 INFO:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:8000/ 2020-11-19T02:52:19.219 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:52:19.219 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:52:19.222 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi086.front.sepia.ceph.com port 8000: Connection refused 2020-11-19T02:52:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T02:52:20.225 INFO:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:8000/ 2020-11-19T02:52:20.254 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:52:20.254 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:52:20.264 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T02:52:20.265 INFO:teuthology.orchestra.run.smithi086.stdout:anonymous 2020-11-19T02:52:20.265 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi086.front.sepia.ceph.com:8001/ 2020-11-19T02:52:20.266 INFO:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:8001/ 2020-11-19T02:52:20.338 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:52:20.339 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:52:20.346 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T02:52:20.350 INFO:teuthology.orchestra.run.smithi086.stdout:anonymous 2020-11-19T02:52:20.351 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi160.front.sepia.ceph.com:8000/ 2020-11-19T02:52:20.352 INFO:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000/ 2020-11-19T02:52:20.384 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:52:20.385 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:52:20.390 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T02:52:20.391 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2020-11-19T02:52:20.391 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi160.front.sepia.ceph.com:8001/ 2020-11-19T02:52:20.392 INFO:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001/ 2020-11-19T02:52:20.449 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:52:20.450 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:52:20.455 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-19T02:52:20.456 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2020-11-19T02:52:20.457 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-11-19T02:52:21.073 INFO:tasks.rgw_multisite:creating realm.. 2020-11-19T02:52:21.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:21.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:21.074 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:21.111 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:21.111 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:21.161 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:52:21.161 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:21.162 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-realm", 2020-11-19T02:52:21.162 INFO:teuthology.orchestra.run.smithi086.stdout: "current_period": "f9590cfc-c7c9-4178-aefb-3fbb666de9c8", 2020-11-19T02:52:21.162 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1 2020-11-19T02:52:21.162 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:52:21.163 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-11-19T02:52:21.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:21.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:21.165 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:21.263+0000 7fcc8883b0c0 -1 old format 2020-11-19T02:52:21.283 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:52:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "test-zonegroup", 2020-11-19T02:52:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zonegroup", 2020-11-19T02:52:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "", 2020-11-19T02:52:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "true", 2020-11-19T02:52:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T02:52:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T02:52:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T02:52:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T02:52:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "", 2020-11-19T02:52:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [], 2020-11-19T02:52:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [], 2020-11-19T02:52:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "", 2020-11-19T02:52:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:21.289 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T02:52:21.289 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T02:52:21.290 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:21.290 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:52:21.292 INFO:tasks.rgw_multisite:creating master zone.. 2020-11-19T02:52:21.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--endpoints', 'http://smithi086.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:21.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--endpoints', 'http://smithi086.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:21.295 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key F8H8OCXZACJSYTWP --secret W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS --endpoints http://smithi086.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:21.385 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:21.386 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:21.464 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:52:21.465 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:21.466 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T02:52:21.466 INFO:teuthology.orchestra.run.smithi086.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-19T02:52:21.466 INFO:teuthology.orchestra.run.smithi086.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-19T02:52:21.467 INFO:teuthology.orchestra.run.smithi086.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-19T02:52:21.467 INFO:teuthology.orchestra.run.smithi086.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-19T02:52:21.467 INFO:teuthology.orchestra.run.smithi086.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-19T02:52:21.468 INFO:teuthology.orchestra.run.smithi086.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-19T02:52:21.468 INFO:teuthology.orchestra.run.smithi086.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-19T02:52:21.468 INFO:teuthology.orchestra.run.smithi086.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-19T02:52:21.468 INFO:teuthology.orchestra.run.smithi086.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-19T02:52:21.468 INFO:teuthology.orchestra.run.smithi086.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-19T02:52:21.469 INFO:teuthology.orchestra.run.smithi086.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-19T02:52:21.469 INFO:teuthology.orchestra.run.smithi086.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-19T02:52:21.469 INFO:teuthology.orchestra.run.smithi086.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-19T02:52:21.469 INFO:teuthology.orchestra.run.smithi086.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-19T02:52:21.469 INFO:teuthology.orchestra.run.smithi086.stdout: "system_key": { 2020-11-19T02:52:21.470 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T02:52:21.470 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T02:52:21.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:21.470 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_pools": [ 2020-11-19T02:52:21.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:21.471 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "default-placement", 2020-11-19T02:52:21.471 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:52:21.471 INFO:teuthology.orchestra.run.smithi086.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-19T02:52:21.471 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": { 2020-11-19T02:52:21.472 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD": { 2020-11-19T02:52:21.472 INFO:teuthology.orchestra.run.smithi086.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2020-11-19T02:52:21.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:21.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:21.472 INFO:teuthology.orchestra.run.smithi086.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-19T02:52:21.473 INFO:teuthology.orchestra.run.smithi086.stdout: "index_type": 0 2020-11-19T02:52:21.473 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:21.473 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:21.473 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:21.473 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:21.474 INFO:teuthology.orchestra.run.smithi086.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-19T02:52:21.474 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:52:21.475 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-11-19T02:52:21.995 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-11-19T02:52:22.013 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-11-19T02:52:22.989 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-11-19T02:52:23.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:23.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:23.008 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:23.061 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:23.061 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:23.081 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:52:23.081 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:23.082 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T02:52:23.082 INFO:teuthology.orchestra.run.smithi086.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-19T02:52:23.083 INFO:teuthology.orchestra.run.smithi086.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-19T02:52:23.083 INFO:teuthology.orchestra.run.smithi086.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-19T02:52:23.083 INFO:teuthology.orchestra.run.smithi086.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-19T02:52:23.084 INFO:teuthology.orchestra.run.smithi086.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-19T02:52:23.084 INFO:teuthology.orchestra.run.smithi086.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-19T02:52:23.084 INFO:teuthology.orchestra.run.smithi086.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-19T02:52:23.084 INFO:teuthology.orchestra.run.smithi086.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-19T02:52:23.084 INFO:teuthology.orchestra.run.smithi086.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-19T02:52:23.085 INFO:teuthology.orchestra.run.smithi086.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-19T02:52:23.085 INFO:teuthology.orchestra.run.smithi086.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-19T02:52:23.085 INFO:teuthology.orchestra.run.smithi086.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-19T02:52:23.085 INFO:teuthology.orchestra.run.smithi086.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-19T02:52:23.085 INFO:teuthology.orchestra.run.smithi086.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-19T02:52:23.086 INFO:teuthology.orchestra.run.smithi086.stdout: "system_key": { 2020-11-19T02:52:23.086 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T02:52:23.086 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T02:52:23.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:23.087 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_pools": [ 2020-11-19T02:52:23.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:23.087 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "default-placement", 2020-11-19T02:52:23.087 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:52:23.087 INFO:teuthology.orchestra.run.smithi086.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-19T02:52:23.088 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": { 2020-11-19T02:52:23.088 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD": { 2020-11-19T02:52:23.088 INFO:teuthology.orchestra.run.smithi086.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-19T02:52:23.088 INFO:teuthology.orchestra.run.smithi086.stdout: "compression_type": "random" 2020-11-19T02:52:23.088 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:23.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:23.089 INFO:teuthology.orchestra.run.smithi086.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-19T02:52:23.089 INFO:teuthology.orchestra.run.smithi086.stdout: "index_type": 0 2020-11-19T02:52:23.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:23.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:23.090 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:23.090 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "", 2020-11-19T02:52:23.090 INFO:teuthology.orchestra.run.smithi086.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-19T02:52:23.090 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:52:23.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:23.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:23.092 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:23.130 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:23.130 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:31.260 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:52:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:52:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2020-11-19T02:52:31.262 INFO:teuthology.orchestra.run.smithi086.stdout: "predecessor_uuid": "f9590cfc-c7c9-4178-aefb-3fbb666de9c8", 2020-11-19T02:52:31.262 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": [], 2020-11-19T02:52:31.262 INFO:teuthology.orchestra.run.smithi086.stdout: "period_map": { 2020-11-19T02:52:31.263 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:52:31.263 INFO:teuthology.orchestra.run.smithi086.stdout: "zonegroups": [ 2020-11-19T02:52:31.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:31.264 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "test-zonegroup", 2020-11-19T02:52:31.264 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zonegroup", 2020-11-19T02:52:31.265 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "", 2020-11-19T02:52:31.265 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "true", 2020-11-19T02:52:31.265 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T02:52:31.265 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T02:52:31.265 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:31.266 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T02:52:31.266 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T02:52:31.266 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:31.267 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [ 2020-11-19T02:52:31.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:31.268 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:31.268 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T02:52:31.268 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T02:52:31.269 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T02:52:31.269 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:31.270 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T02:52:31.270 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "false", 2020-11-19T02:52:31.271 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:52:31.271 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T02:52:31.271 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T02:52:31.272 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T02:52:31.272 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T02:52:31.273 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T02:52:31.273 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:31.273 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:31.274 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [ 2020-11-19T02:52:31.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:31.275 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default-placement", 2020-11-19T02:52:31.275 INFO:teuthology.orchestra.run.smithi086.stdout: "tags": [], 2020-11-19T02:52:31.275 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": [ 2020-11-19T02:52:31.276 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD" 2020-11-19T02:52:31.276 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:52:31.277 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:31.277 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:31.277 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "default-placement", 2020-11-19T02:52:31.278 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:31.278 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T02:52:31.279 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T02:52:31.279 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:31.279 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:31.280 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:31.280 INFO:teuthology.orchestra.run.smithi086.stdout: "short_zone_ids": [ 2020-11-19T02:52:31.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:31.281 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:31.281 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2043733610 2020-11-19T02:52:31.282 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:31.282 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:52:31.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:31.283 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T02:52:31.283 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:31.283 INFO:teuthology.orchestra.run.smithi086.stdout: "period_config": { 2020-11-19T02:52:31.283 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2020-11-19T02:52:31.284 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T02:52:31.284 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T02:52:31.284 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T02:52:31.284 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T02:52:31.284 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T02:52:31.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:31.285 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2020-11-19T02:52:31.285 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T02:52:31.285 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T02:52:31.285 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T02:52:31.286 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T02:52:31.286 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T02:52:31.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:31.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:31.286 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:31.287 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_name": "test-realm", 2020-11-19T02:52:31.287 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:52:31.287 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:52:31.288 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-11-19T02:52:31.288 INFO:teuthology.orchestra.run:waiting for 300 2020-11-19T02:52:31.305 INFO:tasks.rgw.c1.client.0.smithi086.stdout:2020-11-19T02:52:31.303+0000 7fc2c37fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 13158) UID: 0 2020-11-19T02:52:31.306 INFO:tasks.rgw.c1.client.0.smithi086.stdout:2020-11-19T02:52:31.303+0000 7fc2df8c5100 -1 shutting down 2020-11-19T02:52:34.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-11-19T02:52:37.390 INFO:tasks.rgw.c1.client.0:Stopped 2020-11-19T02:52:37.465 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-19T02:52:37.466 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-19T02:52:37.470 INFO:tasks.rgw.c1.client.0:Started 2020-11-19T02:52:37.471 INFO:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:8000 2020-11-19T02:52:37.491 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:52:37.491 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:52:37.496 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi086.front.sepia.ceph.com port 8000: Connection refused 2020-11-19T02:52:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T02:52:38.499 INFO:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:8000 2020-11-19T02:52:38.515 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:52:38.516 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:52:38.524 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T02:52:38.524 INFO:teuthology.orchestra.run.smithi086.stdout:anonymous 2020-11-19T02:52:38.525 INFO:tasks.rgw_multisite:creating admin user.. 2020-11-19T02:52:38.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:38.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:38.527 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key F8H8OCXZACJSYTWP --secret W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:38.618 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:38.619 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:38.832 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:52:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "user_id": "realm-admin", 2020-11-19T02:52:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "display_name": "Realm Admin", 2020-11-19T02:52:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "email": "", 2020-11-19T02:52:38.835 INFO:teuthology.orchestra.run.smithi086.stdout: "suspended": 0, 2020-11-19T02:52:38.836 INFO:teuthology.orchestra.run.smithi086.stdout: "max_buckets": 1000, 2020-11-19T02:52:38.836 INFO:teuthology.orchestra.run.smithi086.stdout: "subusers": [], 2020-11-19T02:52:38.836 INFO:teuthology.orchestra.run.smithi086.stdout: "keys": [ 2020-11-19T02:52:38.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:38.837 INFO:teuthology.orchestra.run.smithi086.stdout: "user": "realm-admin", 2020-11-19T02:52:38.838 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T02:52:38.839 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T02:52:38.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:38.840 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:38.841 INFO:teuthology.orchestra.run.smithi086.stdout: "swift_keys": [], 2020-11-19T02:52:38.841 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": [], 2020-11-19T02:52:38.841 INFO:teuthology.orchestra.run.smithi086.stdout: "op_mask": "read, write, delete", 2020-11-19T02:52:38.841 INFO:teuthology.orchestra.run.smithi086.stdout: "system": "true", 2020-11-19T02:52:38.841 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "", 2020-11-19T02:52:38.842 INFO:teuthology.orchestra.run.smithi086.stdout: "default_storage_class": "", 2020-11-19T02:52:38.842 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_tags": [], 2020-11-19T02:52:38.842 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2020-11-19T02:52:38.842 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T02:52:38.843 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T02:52:38.843 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T02:52:38.843 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T02:52:38.843 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T02:52:38.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:38.844 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2020-11-19T02:52:38.844 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T02:52:38.844 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T02:52:38.844 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T02:52:38.844 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T02:52:38.845 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T02:52:38.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:38.845 INFO:teuthology.orchestra.run.smithi086.stdout: "temp_url_keys": [], 2020-11-19T02:52:38.845 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "rgw", 2020-11-19T02:52:38.845 INFO:teuthology.orchestra.run.smithi086.stdout: "mfa_ids": [] 2020-11-19T02:52:38.847 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:52:38.847 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:52:38.863 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-19T02:52:38.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi086.front.sepia.ceph.com:8000', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:38.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi086.front.sepia.ceph.com:8000', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:38.865 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi086.front.sepia.ceph.com:8000 --access-key F8H8OCXZACJSYTWP --secret W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:38.910 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:38.910 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:39.029 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:52:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-realm", 2020-11-19T02:52:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "current_period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:52:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 2 2020-11-19T02:52:39.033 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:52:39.040 INFO:tasks.rgw_multisite:creating zone.. 2020-11-19T02:52:39.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:39.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:39.051 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key F8H8OCXZACJSYTWP --secret W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS --endpoints http://smithi160.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:39.133 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:39.135 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:39.194 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:39.189+0000 7f6c98e150c0 0 failed reading obj info from .rgw.root:zone_info.d71482d6-15fd-4715-9992-a25958dcb538: (2) No such file or directory 2020-11-19T02:52:39.194 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:52:39.189+0000 7f6c98e150c0 0 WARNING: could not read zone params for zone id=d71482d6-15fd-4715-9992-a25958dcb538 name=test-zone1 2020-11-19T02:52:39.203 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:52:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T02:52:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T02:52:39.205 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-19T02:52:39.207 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-19T02:52:39.207 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-19T02:52:39.207 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-19T02:52:39.208 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-19T02:52:39.208 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-19T02:52:39.208 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-19T02:52:39.209 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-19T02:52:39.209 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-19T02:52:39.210 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-19T02:52:39.210 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-19T02:52:39.210 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-19T02:52:39.211 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-19T02:52:39.211 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-19T02:52:39.211 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T02:52:39.212 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T02:52:39.212 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T02:52:39.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:39.213 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T02:52:39.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:39.213 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T02:52:39.214 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:52:39.214 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-19T02:52:39.215 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T02:52:39.215 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T02:52:39.215 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2020-11-19T02:52:39.216 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:39.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:39.217 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-19T02:52:39.217 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T02:52:39.217 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:39.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:39.218 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:39.218 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:39.219 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-19T02:52:39.219 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:52:39.221 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-11-19T02:52:40.087 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-11-19T02:52:40.103 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-11-19T02:52:40.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-11-19T02:52:41.089 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-11-19T02:52:42.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:42.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:42.563 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:42.608 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:42.608 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:42.648 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:52:42.649 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T02:52:42.649 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T02:52:42.650 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-19T02:52:42.650 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-19T02:52:42.651 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-19T02:52:42.651 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-19T02:52:42.651 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-19T02:52:42.652 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-19T02:52:42.652 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-19T02:52:42.653 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-19T02:52:42.653 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-19T02:52:42.654 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-19T02:52:42.654 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-19T02:52:42.654 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-19T02:52:42.655 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-19T02:52:42.655 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-19T02:52:42.656 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T02:52:42.656 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T02:52:42.656 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T02:52:42.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:42.657 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T02:52:42.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:42.658 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T02:52:42.658 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:52:42.659 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-19T02:52:42.659 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T02:52:42.660 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T02:52:42.660 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-19T02:52:42.660 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2020-11-19T02:52:42.661 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:42.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:42.662 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-19T02:52:42.662 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T02:52:42.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:42.663 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:42.663 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:42.664 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "", 2020-11-19T02:52:42.664 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-19T02:52:42.665 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:52:42.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:42.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:42.668 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:42.747 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:42.748 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:50.283 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone d71482d6-15fd-4715-9992-a25958dcb538 2020-11-19T02:52:50.385 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:52:50.386 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:52:50.387 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 2, 2020-11-19T02:52:50.393 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "f9590cfc-c7c9-4178-aefb-3fbb666de9c8", 2020-11-19T02:52:50.394 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [], 2020-11-19T02:52:50.395 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2020-11-19T02:52:50.396 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:52:50.396 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2020-11-19T02:52:50.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:50.397 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2020-11-19T02:52:50.398 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2020-11-19T02:52:50.398 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2020-11-19T02:52:50.399 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": "true", 2020-11-19T02:52:50.399 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T02:52:50.400 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T02:52:50.400 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:50.401 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2020-11-19T02:52:50.402 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2020-11-19T02:52:50.402 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:50.403 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2020-11-19T02:52:50.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:50.404 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T02:52:50.405 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T02:52:50.406 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T02:52:50.407 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T02:52:50.408 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:50.410 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T02:52:50.411 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T02:52:50.412 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:52:50.413 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T02:52:50.414 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T02:52:50.414 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T02:52:50.415 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T02:52:50.418 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T02:52:50.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:50.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:50.422 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:50.424 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2020-11-19T02:52:50.425 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T02:52:50.427 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T02:52:50.427 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:50.428 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T02:52:50.428 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T02:52:50.428 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:52:50.428 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T02:52:50.428 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T02:52:50.429 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T02:52:50.429 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T02:52:50.429 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T02:52:50.429 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:50.429 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:50.430 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2020-11-19T02:52:50.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:50.430 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2020-11-19T02:52:50.430 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2020-11-19T02:52:50.430 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2020-11-19T02:52:50.431 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2020-11-19T02:52:50.431 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:52:50.431 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:50.431 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:50.431 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2020-11-19T02:52:50.432 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:50.432 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2020-11-19T02:52:50.432 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2020-11-19T02:52:50.432 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:50.432 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:50.433 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:52:50.433 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2020-11-19T02:52:50.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:50.433 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T02:52:50.433 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2049584274 2020-11-19T02:52:50.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:50.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:52:50.434 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:50.434 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2043733610 2020-11-19T02:52:50.434 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:50.435 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:52:50.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:50.435 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T02:52:50.435 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:52:50.435 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2020-11-19T02:52:50.436 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2020-11-19T02:52:50.436 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T02:52:50.436 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T02:52:50.436 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T02:52:50.436 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T02:52:50.437 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T02:52:50.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:50.437 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2020-11-19T02:52:50.437 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T02:52:50.437 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T02:52:50.438 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T02:52:50.438 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T02:52:50.438 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T02:52:50.438 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:52:50.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:52:50.439 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:50.439 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2020-11-19T02:52:50.439 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:52:50.439 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:52:50.440 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-11-19T02:52:50.441 INFO:teuthology.orchestra.run:waiting for 300 2020-11-19T02:52:50.487 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2020-11-19T02:52:50.485+0000 7f7a2886f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 13295) UID: 0 2020-11-19T02:52:50.487 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2020-11-19T02:52:50.485+0000 7f7a3fb0e100 -1 shutting down 2020-11-19T02:52:52.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-11-19T02:52:56.543 INFO:tasks.rgw.c2.client.0:Stopped 2020-11-19T02:52:56.544 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-19T02:52:56.545 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-19T02:52:56.554 INFO:tasks.rgw.c2.client.0:Started 2020-11-19T02:52:56.555 INFO:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2020-11-19T02:52:56.579 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:52:56.580 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:52:56.588 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2020-11-19T02:52:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2020-11-19T02:52:57.605 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:52:57.605 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:52:57.612 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-19T02:52:57.613 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2020-11-19T02:52:57.613 INFO:tasks.rgw_multisite:creating zone.. 2020-11-19T02:52:57.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--endpoints', 'http://smithi086.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:57.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--endpoints', 'http://smithi086.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:52:57.614 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key F8H8OCXZACJSYTWP --secret W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS --endpoints http://smithi086.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:52:57.658 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:52:57.658 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:52:57.713 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:57.711+0000 7f5abaf4e0c0 0 failed reading obj info from .rgw.root:zone_info.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c: (2) No such file or directory 2020-11-19T02:52:57.714 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:52:57.711+0000 7f5abaf4e0c0 0 WARNING: could not read zone params for zone id=c9a90d7d-dd43-4d9d-9b12-b4d640ca300c name=test-zone2 2020-11-19T02:52:57.724 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:52:57.724 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T02:52:57.724 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone3", 2020-11-19T02:52:57.725 INFO:teuthology.orchestra.run.smithi086.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-19T02:52:57.725 INFO:teuthology.orchestra.run.smithi086.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-19T02:52:57.725 INFO:teuthology.orchestra.run.smithi086.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-19T02:52:57.726 INFO:teuthology.orchestra.run.smithi086.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-19T02:52:57.726 INFO:teuthology.orchestra.run.smithi086.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-19T02:52:57.726 INFO:teuthology.orchestra.run.smithi086.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-19T02:52:57.726 INFO:teuthology.orchestra.run.smithi086.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-19T02:52:57.726 INFO:teuthology.orchestra.run.smithi086.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-19T02:52:57.727 INFO:teuthology.orchestra.run.smithi086.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-19T02:52:57.727 INFO:teuthology.orchestra.run.smithi086.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-19T02:52:57.727 INFO:teuthology.orchestra.run.smithi086.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-19T02:52:57.727 INFO:teuthology.orchestra.run.smithi086.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-19T02:52:57.727 INFO:teuthology.orchestra.run.smithi086.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-19T02:52:57.728 INFO:teuthology.orchestra.run.smithi086.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-19T02:52:57.728 INFO:teuthology.orchestra.run.smithi086.stdout: "system_key": { 2020-11-19T02:52:57.728 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T02:52:57.728 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T02:52:57.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:57.729 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_pools": [ 2020-11-19T02:52:57.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:52:57.729 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "default-placement", 2020-11-19T02:52:57.729 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:52:57.729 INFO:teuthology.orchestra.run.smithi086.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-19T02:52:57.730 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": { 2020-11-19T02:52:57.730 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD": { 2020-11-19T02:52:57.730 INFO:teuthology.orchestra.run.smithi086.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2020-11-19T02:52:57.730 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:57.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:52:57.731 INFO:teuthology.orchestra.run.smithi086.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-19T02:52:57.731 INFO:teuthology.orchestra.run.smithi086.stdout: "index_type": 0 2020-11-19T02:52:57.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:57.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:52:57.731 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:52:57.732 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:52:57.732 INFO:teuthology.orchestra.run.smithi086.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-19T02:52:57.732 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:52:57.733 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-11-19T02:52:58.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-11-19T02:52:59.070 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-11-19T02:52:59.089 INFO:teuthology.orchestra.run.smithi086:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-11-19T02:53:00.086 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-11-19T02:53:00.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:00.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:00.105 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:53:00.156 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:00.157 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:00.177 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:53:00.177 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T02:53:00.178 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone3", 2020-11-19T02:53:00.178 INFO:teuthology.orchestra.run.smithi086.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-19T02:53:00.178 INFO:teuthology.orchestra.run.smithi086.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-19T02:53:00.178 INFO:teuthology.orchestra.run.smithi086.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-19T02:53:00.179 INFO:teuthology.orchestra.run.smithi086.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-19T02:53:00.179 INFO:teuthology.orchestra.run.smithi086.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-19T02:53:00.179 INFO:teuthology.orchestra.run.smithi086.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-19T02:53:00.179 INFO:teuthology.orchestra.run.smithi086.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-19T02:53:00.180 INFO:teuthology.orchestra.run.smithi086.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-19T02:53:00.180 INFO:teuthology.orchestra.run.smithi086.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-19T02:53:00.180 INFO:teuthology.orchestra.run.smithi086.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-19T02:53:00.180 INFO:teuthology.orchestra.run.smithi086.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-19T02:53:00.180 INFO:teuthology.orchestra.run.smithi086.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-19T02:53:00.181 INFO:teuthology.orchestra.run.smithi086.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-19T02:53:00.181 INFO:teuthology.orchestra.run.smithi086.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-19T02:53:00.181 INFO:teuthology.orchestra.run.smithi086.stdout: "system_key": { 2020-11-19T02:53:00.181 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T02:53:00.182 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T02:53:00.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:00.182 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_pools": [ 2020-11-19T02:53:00.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:00.188 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "default-placement", 2020-11-19T02:53:00.188 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:53:00.189 INFO:teuthology.orchestra.run.smithi086.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-19T02:53:00.189 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": { 2020-11-19T02:53:00.190 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD": { 2020-11-19T02:53:00.190 INFO:teuthology.orchestra.run.smithi086.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2020-11-19T02:53:00.190 INFO:teuthology.orchestra.run.smithi086.stdout: "compression_type": "random" 2020-11-19T02:53:00.191 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:00.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:00.191 INFO:teuthology.orchestra.run.smithi086.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-19T02:53:00.191 INFO:teuthology.orchestra.run.smithi086.stdout: "index_type": 0 2020-11-19T02:53:00.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:00.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:00.192 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:53:00.193 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "", 2020-11-19T02:53:00.193 INFO:teuthology.orchestra.run.smithi086.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-19T02:53:00.194 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:53:00.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:00.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:00.196 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:53:00.234 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:00.234 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:08.324 INFO:teuthology.orchestra.run.smithi086.stderr:Sending period to new master zone d71482d6-15fd-4715-9992-a25958dcb538 2020-11-19T02:53:08.388 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:53:08.389 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:08.389 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 3, 2020-11-19T02:53:08.389 INFO:teuthology.orchestra.run.smithi086.stdout: "predecessor_uuid": "f9590cfc-c7c9-4178-aefb-3fbb666de9c8", 2020-11-19T02:53:08.389 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": [], 2020-11-19T02:53:08.390 INFO:teuthology.orchestra.run.smithi086.stdout: "period_map": { 2020-11-19T02:53:08.390 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:08.390 INFO:teuthology.orchestra.run.smithi086.stdout: "zonegroups": [ 2020-11-19T02:53:08.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:08.390 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "test-zonegroup", 2020-11-19T02:53:08.391 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zonegroup", 2020-11-19T02:53:08.391 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "", 2020-11-19T02:53:08.391 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "true", 2020-11-19T02:53:08.391 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T02:53:08.391 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T02:53:08.392 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:53:08.392 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T02:53:08.392 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T02:53:08.392 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:53:08.392 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [ 2020-11-19T02:53:08.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:08.393 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T02:53:08.393 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone2", 2020-11-19T02:53:08.393 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T02:53:08.393 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T02:53:08.394 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:53:08.394 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T02:53:08.394 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T02:53:08.394 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:53:08.395 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T02:53:08.395 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T02:53:08.395 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T02:53:08.395 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T02:53:08.395 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T02:53:08.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:08.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:08.396 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:53:08.396 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T02:53:08.396 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T02:53:08.397 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T02:53:08.397 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:53:08.397 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T02:53:08.397 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T02:53:08.397 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:53:08.398 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T02:53:08.398 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T02:53:08.398 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T02:53:08.398 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T02:53:08.398 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T02:53:08.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:08.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:08.399 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T02:53:08.399 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone3", 2020-11-19T02:53:08.400 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T02:53:08.400 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T02:53:08.400 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:53:08.400 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T02:53:08.401 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T02:53:08.401 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:53:08.401 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T02:53:08.401 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T02:53:08.401 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T02:53:08.402 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T02:53:08.402 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T02:53:08.402 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:08.402 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:53:08.402 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [ 2020-11-19T02:53:08.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:08.403 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default-placement", 2020-11-19T02:53:08.403 INFO:teuthology.orchestra.run.smithi086.stdout: "tags": [], 2020-11-19T02:53:08.403 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": [ 2020-11-19T02:53:08.403 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD" 2020-11-19T02:53:08.404 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:53:08.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:08.404 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:53:08.404 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "default-placement", 2020-11-19T02:53:08.404 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:53:08.405 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T02:53:08.405 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T02:53:08.405 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:08.405 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:08.406 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:53:08.406 INFO:teuthology.orchestra.run.smithi086.stdout: "short_zone_ids": [ 2020-11-19T02:53:08.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:08.406 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T02:53:08.406 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2049584274 2020-11-19T02:53:08.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:08.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:08.407 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:53:08.407 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2043733610 2020-11-19T02:53:08.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:08.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:08.408 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T02:53:08.408 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2362875045 2020-11-19T02:53:08.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:08.408 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:53:08.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:08.409 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T02:53:08.409 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:53:08.409 INFO:teuthology.orchestra.run.smithi086.stdout: "period_config": { 2020-11-19T02:53:08.409 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2020-11-19T02:53:08.410 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T02:53:08.410 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T02:53:08.410 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T02:53:08.410 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T02:53:08.410 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T02:53:08.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:08.411 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2020-11-19T02:53:08.411 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T02:53:08.411 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T02:53:08.416 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T02:53:08.416 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T02:53:08.416 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T02:53:08.417 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:08.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:08.417 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:53:08.417 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_name": "test-realm", 2020-11-19T02:53:08.417 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:53:08.418 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:53:08.419 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-19T02:53:08.419 INFO:teuthology.orchestra.run:waiting for 300 2020-11-19T02:53:08.464 INFO:tasks.rgw.c1.client.1.smithi086.stdout:2020-11-19T02:53:08.463+0000 7fc1f77fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 13161) UID: 0 2020-11-19T02:53:08.464 INFO:tasks.rgw.c1.client.1.smithi086.stdout:2020-11-19T02:53:08.463+0000 7fc21b8cf100 -1 shutting down 2020-11-19T02:53:09.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-19T02:53:14.522 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-19T02:53:14.522 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-19T02:53:14.523 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-19T02:53:14.526 INFO:tasks.rgw.c1.client.1:Started 2020-11-19T02:53:14.526 INFO:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:8001 2020-11-19T02:53:14.543 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:53:14.543 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:53:14.547 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi086.front.sepia.ceph.com port 8001: Connection refused 2020-11-19T02:53:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T02:53:14.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2020-11-19T02:53:15.549 INFO:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:8001 2020-11-19T02:53:15.563 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:53:15.563 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:53:15.571 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T02:53:15.571 INFO:teuthology.orchestra.run.smithi086.stdout:anonymous 2020-11-19T02:53:15.572 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-19T02:53:15.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi086.front.sepia.ceph.com:8000', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--url', 'http://smithi086.front.sepia.ceph.com:8000', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:15.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi086.front.sepia.ceph.com:8000', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--url', 'http://smithi086.front.sepia.ceph.com:8000', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:15.573 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi086.front.sepia.ceph.com:8000 --access-key F8H8OCXZACJSYTWP --secret W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS --url http://smithi086.front.sepia.ceph.com:8000 --access-key F8H8OCXZACJSYTWP --secret W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T02:53:15.624 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:15.625 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:15.669 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2020-11-19T02:53:15.665+0000 7ff216e05700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.46bd204f-1547-4756-ae4a-a8264d05dd86.control err=-107 2020-11-19T02:53:15.696 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:53:15.696 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:53:15.696 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-realm", 2020-11-19T02:53:15.696 INFO:teuthology.orchestra.run.smithi160.stdout: "current_period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:15.697 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 2 2020-11-19T02:53:15.697 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:53:15.698 INFO:tasks.rgw_multisite:creating zone.. 2020-11-19T02:53:15.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:15.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:15.702 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key F8H8OCXZACJSYTWP --secret W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS --endpoints http://smithi160.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T02:53:15.754 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:15.755 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:15.816 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:53:15.813+0000 7f884b06b0c0 0 failed reading obj info from .rgw.root:zone_info.d71482d6-15fd-4715-9992-a25958dcb538: (2) No such file or directory 2020-11-19T02:53:15.816 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:53:15.813+0000 7f884b06b0c0 0 WARNING: could not read zone params for zone id=d71482d6-15fd-4715-9992-a25958dcb538 name=test-zone1 2020-11-19T02:53:15.817 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:53:15.813+0000 7f884b06b0c0 0 failed reading obj info from .rgw.root:zone_info.fdf730f5-4390-45ac-9d84-6887e17cb661: (2) No such file or directory 2020-11-19T02:53:15.817 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:53:15.813+0000 7f884b06b0c0 0 WARNING: could not read zone params for zone id=fdf730f5-4390-45ac-9d84-6887e17cb661 name=test-zone3 2020-11-19T02:53:15.824 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:53:15.825 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T02:53:15.825 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2020-11-19T02:53:15.825 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-19T02:53:15.825 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-19T02:53:15.826 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-19T02:53:15.826 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-19T02:53:15.826 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-19T02:53:15.826 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-19T02:53:15.826 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-19T02:53:15.827 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-19T02:53:15.827 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-19T02:53:15.827 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-19T02:53:15.827 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-19T02:53:15.827 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-19T02:53:15.828 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-19T02:53:15.828 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-19T02:53:15.828 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T02:53:15.828 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T02:53:15.829 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T02:53:15.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:15.829 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T02:53:15.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:15.829 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T02:53:15.830 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:15.830 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-19T02:53:15.830 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T02:53:15.830 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T02:53:15.830 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2020-11-19T02:53:15.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:15.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:15.831 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-19T02:53:15.831 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T02:53:15.834 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:15.834 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:15.834 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:15.834 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_config": { 2020-11-19T02:53:15.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event_retention_days": 7, 2020-11-19T02:53:15.835 INFO:teuthology.orchestra.run.smithi160.stdout: "start_with_full_sync": false 2020-11-19T02:53:15.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:15.835 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:53:15.836 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-19T02:53:15.837 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:53:15.838 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-11-19T02:53:16.190 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-11-19T02:53:16.208 INFO:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-11-19T02:53:17.220 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-11-19T02:53:17.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:17.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:17.242 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T02:53:17.288 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:17.290 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:17.311 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:53:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T02:53:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2020-11-19T02:53:17.312 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-19T02:53:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-19T02:53:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-19T02:53:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-19T02:53:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-19T02:53:17.313 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-19T02:53:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-19T02:53:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-19T02:53:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-19T02:53:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-19T02:53:17.314 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-19T02:53:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-19T02:53:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-19T02:53:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-19T02:53:17.315 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T02:53:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T02:53:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T02:53:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T02:53:17.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T02:53:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-19T02:53:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T02:53:17.317 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T02:53:17.318 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2020-11-19T02:53:17.319 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2020-11-19T02:53:17.321 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:17.323 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-19T02:53:17.324 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T02:53:17.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:17.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:17.327 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:17.328 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_config": { 2020-11-19T02:53:17.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event_retention_days": 7, 2020-11-19T02:53:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: "start_with_full_sync": false 2020-11-19T02:53:17.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "", 2020-11-19T02:53:17.331 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-19T02:53:17.332 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:53:17.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:17.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:17.335 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T02:53:17.419 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:17.419 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:20.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-19T02:53:25.414 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone d71482d6-15fd-4715-9992-a25958dcb538 2020-11-19T02:53:25.486 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:53:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 4, 2020-11-19T02:53:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "f9590cfc-c7c9-4178-aefb-3fbb666de9c8", 2020-11-19T02:53:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [], 2020-11-19T02:53:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2020-11-19T02:53:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2020-11-19T02:53:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2020-11-19T02:53:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2020-11-19T02:53:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2020-11-19T02:53:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": "true", 2020-11-19T02:53:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T02:53:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T02:53:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2020-11-19T02:53:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2020-11-19T02:53:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:53:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2020-11-19T02:53:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T02:53:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2020-11-19T02:53:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T02:53:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T02:53:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T02:53:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T02:53:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:53:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T02:53:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2020-11-19T02:53:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "false", 2020-11-19T02:53:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2020-11-19T02:53:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2020-11-19T02:53:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T02:53:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T02:53:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T02:53:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T02:53:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T02:53:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T02:53:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T02:53:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:53:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T02:53:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T02:53:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T02:53:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T02:53:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T02:53:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:26.266 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:53:26.267 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2020-11-19T02:53:26.267 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T02:53:26.267 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T02:53:26.268 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:26.268 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T02:53:26.269 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T02:53:26.269 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:53:26.269 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T02:53:26.270 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T02:53:26.270 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T02:53:26.270 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T02:53:26.270 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T02:53:26.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:26.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:26.271 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T02:53:26.271 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2020-11-19T02:53:26.271 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T02:53:26.271 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T02:53:26.272 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:26.272 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T02:53:26.272 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T02:53:26.272 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T02:53:26.272 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T02:53:26.273 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T02:53:26.273 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T02:53:26.273 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T02:53:26.273 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T02:53:26.274 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:26.274 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:26.274 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2020-11-19T02:53:26.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:26.274 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2020-11-19T02:53:26.275 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2020-11-19T02:53:26.275 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2020-11-19T02:53:26.275 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2020-11-19T02:53:26.275 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:53:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2020-11-19T02:53:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:53:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2020-11-19T02:53:26.277 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2020-11-19T02:53:26.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:26.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:26.277 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T02:53:26.278 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2020-11-19T02:53:26.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:26.279 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T02:53:26.279 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 1446762810 2020-11-19T02:53:26.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:26.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:26.280 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T02:53:26.280 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2049584274 2020-11-19T02:53:26.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:26.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:26.281 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:53:26.281 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2043733610 2020-11-19T02:53:26.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:26.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:26.282 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T02:53:26.282 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2362875045 2020-11-19T02:53:26.282 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:26.282 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:53:26.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:26.283 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T02:53:26.283 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T02:53:26.283 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2020-11-19T02:53:26.283 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2020-11-19T02:53:26.284 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T02:53:26.284 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T02:53:26.284 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T02:53:26.284 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T02:53:26.285 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T02:53:26.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:26.285 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2020-11-19T02:53:26.285 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T02:53:26.285 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T02:53:26.286 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T02:53:26.286 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T02:53:26.286 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T02:53:26.286 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:26.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:26.287 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T02:53:26.287 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2020-11-19T02:53:26.287 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:26.287 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:53:26.288 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-11-19T02:53:26.288 INFO:teuthology.orchestra.run:waiting for 300 2020-11-19T02:53:26.327 INFO:tasks.rgw.c2.client.1.smithi160.stdout:2020-11-19T02:53:26.325+0000 7fef7171b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 13301) UID: 0 2020-11-19T02:53:26.327 INFO:tasks.rgw.c2.client.1.smithi160.stdout:2020-11-19T02:53:26.325+0000 7fef891bb100 -1 shutting down 2020-11-19T02:53:26.883 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-11-19T02:53:32.390 INFO:tasks.rgw.c2.client.1:Stopped 2020-11-19T02:53:33.226 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-19T02:53:33.227 INFO:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-19T02:53:33.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~6s 2020-11-19T02:53:33.231 INFO:tasks.rgw.c2.client.1:Started 2020-11-19T02:53:33.232 INFO:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2020-11-19T02:53:33.249 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:53:33.250 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:53:33.253 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001: Connection refused 2020-11-19T02:53:33.254 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T02:53:34.256 INFO:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2020-11-19T02:53:34.272 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T02:53:34.273 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2020-11-19T02:53:34.277 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-11-19T02:53:34.278 INFO:teuthology.orchestra.run.smithi160.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005fb5de2e-10e7-test-zone4","HostId":"10e7-test-zone4-test-zonegroup"} 2020-11-19T02:53:34.278 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-11-19T02:53:34.278 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-11-19T02:53:34.356 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-11-19T02:53:34.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:34.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T02:53:34.358 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T02:53:34.403 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:34.403 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:34.647 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:53:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "user_id": "rgw-multisite-test-user", 2020-11-19T02:53:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "display_name": "Multisite Test User", 2020-11-19T02:53:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "email": "", 2020-11-19T02:53:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "suspended": 0, 2020-11-19T02:53:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "max_buckets": 1000, 2020-11-19T02:53:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "subusers": [], 2020-11-19T02:53:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "keys": [ 2020-11-19T02:53:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "user": "rgw-multisite-test-user", 2020-11-19T02:53:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "ENI06Y2F5MOYE0Z9VUP3", 2020-11-19T02:53:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "d8XoyxLocWJtQ58nZKhKkVmDIWLuSCiUWWCzaiEl" 2020-11-19T02:53:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:34.651 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T02:53:34.651 INFO:teuthology.orchestra.run.smithi086.stdout: "swift_keys": [], 2020-11-19T02:53:34.651 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": [], 2020-11-19T02:53:34.651 INFO:teuthology.orchestra.run.smithi086.stdout: "op_mask": "read, write, delete", 2020-11-19T02:53:34.651 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "", 2020-11-19T02:53:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "default_storage_class": "", 2020-11-19T02:53:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_tags": [], 2020-11-19T02:53:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2020-11-19T02:53:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T02:53:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T02:53:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T02:53:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T02:53:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T02:53:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2020-11-19T02:53:34.654 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T02:53:34.654 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T02:53:34.654 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T02:53:34.654 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T02:53:34.654 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T02:53:34.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:34.655 INFO:teuthology.orchestra.run.smithi086.stdout: "temp_url_keys": [], 2020-11-19T02:53:34.655 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "rgw", 2020-11-19T02:53:34.655 INFO:teuthology.orchestra.run.smithi086.stdout: "mfa_ids": [] 2020-11-19T02:53:34.655 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:53:34.656 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T02:53:34.677 INFO:rgw_multi.tests:meta checkpoint 2020-11-19T02:53:34.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:34.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:34.679 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:53:34.737 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:34.738 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:34.952 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:53:34.978 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:53:34.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:34.979 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:53:34.979 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:53:34.640760Z" 2020-11-19T02:53:34.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:34.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:34.980 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:53:34.980 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:53:34.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:34.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:34.981 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:53:34.981 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:53:34.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:53:34.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:53:34.982 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:53:34.982 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:53:34.982 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:53:34.982 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:53:34.983 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:53:34.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:34.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:34.985 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:53:35.027 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:35.028 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:35.289 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:53:35.289 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:53:35.290 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:53:35.290 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:53:35.290 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:53:35.290 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:35.291 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:35.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:35.291 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:53:35.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:35.291 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:53:35.292 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:35.292 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:35.292 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:35.292 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:35.293 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:53:35.293 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:35.293 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:35.293 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:35.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:35.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:35.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:35.294 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:53:35.294 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:35.295 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:35.295 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:35.295 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:35.295 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:35.295 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:35.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:35.296 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:35.296 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:35.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:35.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:35.297 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:53:35.297 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:35.297 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:35.297 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:35.298 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:35.298 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:35.298 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:35.298 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:35.298 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:35.299 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:35.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:35.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:35.299 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:53:35.300 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:35.300 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:35.300 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:35.300 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:35.300 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:35.301 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:35.301 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:35.301 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:35.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:35.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:35.302 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:53:35.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:35.302 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:53:35.302 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:53:35.303 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:53:35.303 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:35.303 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:53:35.304 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:53:38.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-11-19T02:53:40.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:40.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:40.306 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:53:40.348 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:40.348 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:40.604 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:53:40.605 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:53:40.605 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:53:40.605 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:53:40.605 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:53:40.606 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:40.606 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:40.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:40.606 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:53:40.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:40.607 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:53:40.607 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:40.607 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:40.607 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:40.608 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:40.608 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:53:40.608 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:40.608 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:40.609 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:40.609 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:40.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:40.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:40.609 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:53:40.610 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:40.610 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:40.610 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:40.610 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:40.611 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:40.611 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:40.611 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:40.611 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:40.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:40.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:40.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:40.612 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:53:40.612 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:40.612 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:40.613 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:40.613 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:40.613 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:40.613 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:40.614 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:40.614 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:40.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:40.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:40.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:40.615 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:53:40.615 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:40.615 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:40.615 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:40.615 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:40.616 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:40.616 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:40.616 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:40.616 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:40.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:40.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:40.617 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:53:40.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:40.617 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:53:40.618 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:53:40.618 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:53:40.618 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:40.618 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:53:40.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:53:45.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:45.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:45.622 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:53:45.661 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:45.662 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:45.922 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:53:45.923 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:53:45.923 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:53:45.923 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:53:45.924 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:53:45.924 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:45.924 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:45.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:45.924 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:53:45.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:45.925 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:53:45.925 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:45.925 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:45.925 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:45.926 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:45.926 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:53:45.926 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:45.926 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:45.927 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:45.927 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:45.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:45.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:45.927 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:53:45.928 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:45.928 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:45.928 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:45.928 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:45.928 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:45.929 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:45.929 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:45.929 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:45.929 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:45.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:45.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:45.930 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:53:45.930 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:45.930 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:45.931 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:45.931 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:45.931 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:45.931 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:45.931 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:45.932 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:45.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:45.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:45.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:45.933 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:53:45.933 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:45.933 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:45.933 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:45.934 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:45.934 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:45.934 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:45.934 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:45.934 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:45.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:45.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:45.935 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:53:45.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:45.935 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:53:45.936 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:53:45.936 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:53:45.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:45.936 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:53:45.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:53:50.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:50.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:50.940 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:53:50.998 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:50.998 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:51.253 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:53:51.254 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:53:51.255 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:53:51.256 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:53:51.256 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:53:51.257 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:51.271 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:51.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:51.273 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:53:51.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:51.275 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:53:51.276 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:51.279 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:51.285 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:51.288 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:51.293 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:53:51.293 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:51.294 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:51.294 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:51.294 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:51.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:51.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:51.295 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:53:51.295 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:51.295 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:51.295 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:51.296 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:51.296 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:51.296 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:51.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:51.297 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:51.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:51.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:51.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:51.298 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:53:51.298 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:51.298 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:51.298 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:51.298 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:51.299 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:51.299 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:51.299 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:51.299 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:51.299 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:51.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:51.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:51.300 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:53:51.300 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:51.300 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:51.301 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:51.301 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:51.301 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:51.301 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:51.305 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:51.305 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:51.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:51.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:51.306 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:53:51.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:51.306 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:53:51.306 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:53:51.307 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:53:51.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:51.307 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:53:51.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:53:56.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:56.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:53:56.310 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:53:56.377 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:53:56.377 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:53:56.618 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:53:56.618 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:53:56.619 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:53:56.619 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:53:56.619 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:53:56.619 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:53:56.620 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:56.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:56.620 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:53:56.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:56.620 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:53:56.621 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:56.621 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:56.621 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:56.621 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:56.621 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:53:56.622 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:56.622 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:56.622 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:56.622 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:56.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:56.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:56.623 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:53:56.623 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:56.623 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:56.623 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:56.624 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:56.624 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:56.624 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:56.624 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:56.625 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:56.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:56.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:56.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:56.625 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:53:56.626 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:56.626 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:56.626 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:56.626 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:56.626 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:56.627 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:56.627 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:56.627 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:56.627 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:56.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:56.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:53:56.628 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:53:56.628 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:53:56.628 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:53:56.628 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:53:56.629 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:53:56.629 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:53:56.629 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:53:56.629 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:53:56.629 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:53:56.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:56.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:56.639 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:53:56.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:53:56.640 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:53:56.640 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:53:56.641 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:53:56.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:53:56.641 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:53:56.643 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:01.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:01.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:01.646 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:01.693 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:01.694 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:01.940 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:01.953 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:01.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:06.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:07.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:07.326 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:07.368 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:07.368 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:07.622 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:07.622 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:07.623 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:07.623 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:07.623 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:07.623 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:07.624 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:07.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:07.624 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:07.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:07.625 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:07.625 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:07.625 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:07.625 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:07.626 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:07.626 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:07.626 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:07.627 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:07.627 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:07.627 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:07.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:07.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:07.628 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:07.628 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:07.628 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:07.629 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:07.629 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:07.629 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:07.629 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:07.630 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:07.630 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:07.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:07.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:07.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:07.631 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:07.632 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:07.632 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:07.632 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:07.632 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:07.633 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:07.633 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:07.633 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:07.633 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:07.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:07.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:07.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:07.634 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:07.635 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:07.635 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:07.635 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:07.636 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:07.636 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:07.636 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:07.637 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:07.637 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:07.637 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:07.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:07.638 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:07.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:07.639 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:07.639 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:07.640 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:07.640 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:07.640 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:07.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:12.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:12.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:12.865 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:12.909 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:12.910 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:13.157 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:13.158 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:13.158 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:13.158 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:13.158 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:13.158 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:13.159 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:13.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:13.159 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:13.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:13.160 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:13.160 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:13.160 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:13.160 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:13.160 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:13.161 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:13.161 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:13.161 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:13.161 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:13.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:13.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:13.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:13.162 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:13.162 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:13.162 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:13.163 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:13.163 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:13.163 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:13.163 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:13.163 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:13.164 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:13.164 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:13.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:13.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:13.168 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:13.169 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:13.169 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:13.169 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:13.169 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:13.170 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:13.170 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:13.170 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:13.170 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:13.170 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:13.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:13.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:13.171 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:13.171 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:13.171 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:13.172 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:13.172 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:13.172 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:13.172 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:13.172 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:13.173 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:13.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:13.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:13.173 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:13.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:13.174 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:13.174 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:13.174 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:13.174 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:13.175 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:13.176 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:18.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:18.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:18.179 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:18.221 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:18.221 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:18.463 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:18.463 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:18.464 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:18.464 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:18.465 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:18.465 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:18.465 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:18.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:18.466 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:18.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:18.467 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:18.467 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:18.467 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:18.468 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:18.468 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:18.468 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:18.469 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:18.469 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:18.469 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:18.470 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:18.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:18.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:18.471 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:18.471 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:18.471 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:18.472 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:18.472 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:18.473 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:18.473 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:18.473 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:18.474 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:18.474 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:18.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:18.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:18.476 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:18.476 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:18.476 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:18.477 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:18.478 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:18.478 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:18.479 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:18.479 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:18.479 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:18.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:18.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:18.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:18.481 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:18.481 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:18.482 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:18.483 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:18.483 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:18.483 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:18.484 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:18.484 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:18.484 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:18.485 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:18.485 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:18.486 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:18.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:18.486 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:18.487 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:18.487 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:18.488 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:18.488 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:18.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:23.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:23.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:23.491 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:23.530 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:23.530 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:23.780 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:23.780 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:23.781 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:23.781 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:23.781 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:23.781 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:23.782 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:23.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:23.782 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:23.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:23.782 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:23.783 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:23.783 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:23.783 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:23.783 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:23.783 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:23.784 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:23.784 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:23.784 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:23.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:23.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:23.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:23.785 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:23.785 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:23.785 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:23.786 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:23.786 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:23.786 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:23.786 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:23.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:23.787 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:23.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:23.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:23.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:23.788 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:23.788 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:23.788 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:23.788 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:23.789 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:23.789 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:23.789 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:23.790 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:23.790 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:23.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:23.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:23.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:23.792 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:23.792 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:23.792 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:23.799 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:23.800 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:28.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:28.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:28.803 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:28.843 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:28.845 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:29.102 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:29.103 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:29.103 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:29.103 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:29.103 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:29.104 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:29.104 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:29.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:29.104 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:29.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:29.105 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:29.105 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:29.105 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:29.105 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:29.106 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:29.106 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:29.106 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:29.106 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:29.106 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:29.107 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:29.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:29.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:29.107 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:29.108 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:29.108 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:29.108 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:29.108 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:29.108 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:29.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:29.110 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:29.110 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:29.110 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:29.110 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:29.111 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:29.111 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:29.111 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:29.112 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:29.112 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:29.112 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:29.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:29.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:29.113 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:29.113 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:29.113 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:29.114 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:29.114 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:29.114 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:29.114 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:29.114 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:29.115 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:29.115 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:29.115 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:29.115 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:29.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:29.116 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:29.116 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:29.116 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:29.116 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:29.116 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:29.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:34.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:34.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:34.120 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:34.166 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:34.167 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:34.395 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:34.401 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:34.401 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:34.401 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:34.410 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:34.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:39.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:39.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:39.413 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:39.454 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:39.455 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:39.694 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:39.695 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:39.695 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:39.695 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:39.696 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:39.696 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:39.696 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:39.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:39.697 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:39.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:39.697 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:39.698 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:39.698 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:39.698 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:39.699 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:39.699 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:39.699 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:39.700 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:39.700 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:39.700 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:39.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:39.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:39.701 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:39.701 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:39.702 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:39.702 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:39.702 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:39.703 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:39.703 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:39.703 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:39.704 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:39.704 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:39.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:39.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:39.705 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:39.705 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:39.706 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:39.706 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:39.706 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:39.706 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:39.707 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:39.707 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:39.707 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:39.708 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:39.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:39.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:39.709 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:39.709 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:39.709 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:39.709 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:39.710 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:39.710 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:39.710 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:39.711 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:39.711 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:39.711 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:39.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:39.712 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:39.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:39.712 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:39.713 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:39.713 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:39.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:39.714 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:39.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:44.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:45.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:45.320 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:45.366 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:45.367 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:45.634 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:45.634 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:45.635 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:45.635 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:45.635 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:45.635 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:45.636 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:45.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:45.636 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:45.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:45.636 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:45.637 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:45.637 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:45.637 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:45.637 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:45.637 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:45.638 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:45.638 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:45.638 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:45.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:45.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:45.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:45.639 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:45.639 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:45.639 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:45.639 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:45.640 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:45.640 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:45.640 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:45.640 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:45.640 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:45.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:45.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:45.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:45.641 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:45.642 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:45.642 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:45.642 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:45.642 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:45.642 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:45.643 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:45.643 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:45.643 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:45.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:45.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:45.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:45.644 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:45.644 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:45.644 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:45.644 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:45.645 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:45.645 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:45.645 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:45.645 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:45.645 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:45.646 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:45.646 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:45.646 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:45.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:45.647 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:45.647 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:45.647 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:45.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:45.648 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:45.649 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:50.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:50.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:50.651 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:50.694 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:50.695 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:50.946 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:50.946 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:50.947 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:50.947 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:50.947 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:50.947 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:50.948 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:50.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:50.948 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:50.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:50.948 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:50.949 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:50.949 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:50.949 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:50.949 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:50.950 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:50.950 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:50.950 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:50.950 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:50.950 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:50.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:50.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:50.951 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:50.951 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:50.951 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:50.952 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:50.952 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:50.952 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:50.953 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:50.953 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:50.954 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:50.955 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:50.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:50.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:50.955 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:50.956 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:50.956 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:50.956 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:50.956 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:50.956 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:50.957 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:50.957 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:50.957 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:50.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:50.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:50.958 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:50.958 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:50.958 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:50.958 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:50.958 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:50.959 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:50.959 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:50.959 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:50.959 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:50.960 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:50.960 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:50.960 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:50.960 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:50.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:50.961 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:50.961 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:50.961 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:50.961 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:50.961 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:50.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:54:55.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:55.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:54:55.965 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:54:56.009 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:54:56.010 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:54:56.255 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:54:56.255 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:54:56.256 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:54:56.257 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:54:56.257 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:54:56.258 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:54:56.259 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:56.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:56.260 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:54:56.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:56.261 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:54:56.262 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:56.262 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:56.263 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:56.264 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:56.265 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:54:56.265 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:56.266 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:56.267 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:56.267 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:56.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:56.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:56.269 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:54:56.270 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:56.271 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:56.271 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:56.272 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:56.274 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:56.274 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:56.274 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:56.275 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:56.275 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:56.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:56.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:56.276 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:54:56.276 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:56.276 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:56.276 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:56.276 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:56.277 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:56.277 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:56.277 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:56.277 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:56.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:56.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:56.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:54:56.278 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:54:56.278 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:54:56.279 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:54:56.279 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:54:56.279 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:54:56.279 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:54:56.279 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:54:56.280 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:54:56.280 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:54:56.280 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:56.280 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:56.280 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:54:56.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:54:56.281 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:54:56.281 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:54:56.281 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:54:56.282 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:54:56.282 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:54:56.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:55:01.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:01.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:01.284 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:01.328 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:01.329 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:01.604 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:01.604 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:01.605 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:01.605 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:01.605 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:01.605 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:01.606 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:01.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:01.606 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:01.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:01.606 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:01.607 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:01.607 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:01.607 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:01.607 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:01.607 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:01.608 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:01.608 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:01.608 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:01.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:01.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:01.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:01.609 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:01.609 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:01.609 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:01.610 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:01.610 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:01.610 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:01.610 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:01.610 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:01.611 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:01.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:01.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:01.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:01.611 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:01.612 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:01.612 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:01.612 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:01.612 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:01.612 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:01.613 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:01.613 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:01.613 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:01.613 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:01.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:01.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:01.614 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:01.614 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:01.614 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:01.615 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:01.615 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:01.615 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:01.615 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:01.615 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:01.616 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:01.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:01.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:01.616 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:01.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:01.617 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:01.617 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:01.617 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:01.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:01.618 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:01.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:55:06.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:07.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:07.754 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:07.794 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:07.795 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:08.047 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:08.048 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:08.048 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:08.048 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:08.048 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:08.049 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:08.049 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:08.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:08.049 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:08.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:08.050 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:08.050 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:08.050 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:08.050 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:08.051 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:08.051 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:08.051 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:08.051 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:08.051 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:08.052 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:08.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:08.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:08.052 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:08.052 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:08.053 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:08.053 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:08.053 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:08.053 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:08.053 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:08.054 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:08.054 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:08.054 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:08.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:08.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:08.055 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:08.055 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:08.055 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:08.055 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:08.055 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:08.056 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:08.056 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:08.056 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:08.056 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:08.056 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:08.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:08.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:08.057 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:08.057 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:08.057 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:08.058 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:08.058 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:08.058 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:08.058 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:08.058 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:08.059 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:08.059 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:08.059 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:08.059 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:08.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:08.060 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:08.060 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:08.060 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:08.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:08.061 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:08.062 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:55:13.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:13.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:13.064 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:13.358 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:13.359 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:13.359 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:13.360 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:13.360 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:13.360 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:13.361 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:13.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:13.361 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:13.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:13.362 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:13.362 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:13.362 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:13.362 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:13.362 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:13.363 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:13.363 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:13.363 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:13.363 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:13.364 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:13.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:13.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:13.364 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:13.365 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:13.365 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:13.365 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:13.365 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:13.366 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:13.367 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:13.367 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:13.367 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:13.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:13.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:13.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:13.368 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:13.368 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:13.369 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:13.369 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:13.369 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:13.369 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:13.369 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:13.370 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:13.370 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:13.370 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:13.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:13.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:13.371 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:13.371 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:13.371 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:13.372 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:13.372 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:13.372 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:13.372 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:13.372 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:13.373 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:13.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:13.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:13.373 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:13.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:13.374 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:13.374 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:13.374 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:13.375 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:13.375 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:13.376 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:55:18.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:18.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:18.379 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:18.421 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:18.422 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:18.685 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:18.686 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:18.686 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:18.687 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:18.687 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:18.688 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:18.688 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:18.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:18.689 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:18.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:18.689 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:18.690 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:18.690 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:18.690 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:18.690 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:18.691 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:18.691 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:18.691 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:18.691 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:18.692 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:18.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:18.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:18.692 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:18.692 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:18.693 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:18.693 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:18.693 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:18.693 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:18.694 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:18.694 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:18.694 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:18.694 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:18.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:18.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:18.695 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:18.695 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:18.695 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:18.695 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:18.696 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:18.696 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:18.696 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:18.696 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:18.696 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:18.697 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:18.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:18.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:18.697 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:18.697 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:18.698 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:18.698 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:18.698 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:18.698 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:18.698 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:18.699 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:18.699 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:18.699 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:18.699 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:18.699 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:18.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:18.700 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:18.700 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:18.700 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:18.700 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:18.701 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:18.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:55:23.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:23.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:23.703 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:23.746 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:23.747 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:24.006 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:24.007 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:24.007 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:24.008 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:24.008 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:24.008 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:24.009 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:24.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:24.010 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:24.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:24.010 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:24.011 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:24.011 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:24.012 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:24.012 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:24.012 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:24.013 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:24.013 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:24.014 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:24.014 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:24.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:24.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:24.015 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:24.015 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:24.016 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:24.016 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:24.016 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:24.017 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:24.017 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:24.017 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:24.018 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:24.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:24.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:24.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:24.019 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:24.020 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:24.020 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:24.020 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:24.021 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:24.021 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:24.022 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:24.022 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:24.022 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:24.023 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:24.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:24.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:24.024 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:24.024 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:24.024 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:24.025 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:24.025 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:24.025 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:24.026 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:24.026 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:24.026 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:24.027 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:24.027 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:24.028 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:24.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:24.028 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:24.029 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:24.029 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:24.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:24.030 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:24.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605754414.640760_109.1 target= 2020-11-19T02:55:29.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:29.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:29.034 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:29.086 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:29.087 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:29.334 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:29.335 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:29.335 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:29.335 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:29.335 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:29.337 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:29.337 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:29.337 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:29.337 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:29.338 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:29.338 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:29.338 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:29.338 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:29.338 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:29.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:29.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:29.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:29.339 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:29.339 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:29.340 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:29.340 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:29.340 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:29.340 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:29.340 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:29.341 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:29.341 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:29.341 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:29.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:29.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:29.342 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:29.342 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:29.342 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:29.342 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:29.342 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:29.343 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:29.343 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:29.343 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:29.343 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:29.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:29.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:29.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:29.344 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:29.344 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:29.344 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:29.345 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:29.345 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:29.345 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:29.345 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:29.345 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:29.346 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:29.346 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:29.346 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:29.346 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:29.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:29.347 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:29.347 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:29.347 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:29.347 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:29.347 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:29.349 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:55:29.349 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:55:29.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:29.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:29.350 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:29.393 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:29.394 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:29.638 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:55:29.638 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:55:29.639 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:55:29.639 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:55:29.639 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:55:29.640 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:29.640 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:29.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:29.643 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:29.643 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:29.644 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:55:29.644 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:29.644 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:29.645 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:29.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:29.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:29.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:29.646 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:55:29.647 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:29.647 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:29.648 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:29.648 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:29.648 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:29.649 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:29.649 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:29.649 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:55:29.657 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:29.657 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:29.657 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:29.658 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:29.658 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:29.659 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:29.659 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:29.659 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:29.660 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:29.660 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:29.661 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:55:29.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:29.661 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:55:29.662 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:55:29.662 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:55:29.663 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:29.663 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:55:29.665 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:55:29.666 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:55:29.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:29.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:29.667 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:29.709 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:29.709 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:30.073 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:30.074 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:30.074 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:30.074 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:30.075 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:30.075 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:30.075 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.075 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:30.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:30.076 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:30.076 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:30.076 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:30.077 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:30.077 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:30.077 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:30.077 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:30.077 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:30.078 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:30.079 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:30.079 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:30.079 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:30.079 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:30.079 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:30.080 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:30.080 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:30.080 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:30.080 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:30.081 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:30.081 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:30.081 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:30.082 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:30.082 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:30.082 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:30.082 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:30.082 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:30.083 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.083 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:30.084 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:30.084 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:30.084 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:30.084 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:30.085 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:30.085 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:30.085 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:30.086 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:30.086 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.087 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:30.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.088 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:30.088 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:30.089 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:30.089 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.090 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:30.091 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:55:30.092 INFO:rgw_multi.tests:meta checkpoint 2020-11-19T02:55:30.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:30.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:30.094 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:30.147 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:30.364 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:55:30.397 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:55:30.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:30.398 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:30.398 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:53:34.640760Z" 2020-11-19T02:55:30.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:30.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:30.399 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:30.400 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:30.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:30.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:30.401 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:30.401 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:30.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:30.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:30.402 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:30.403 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:30.403 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:30.404 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:55:30.405 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:55:30.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:30.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:30.407 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:30.450 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:30.451 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:30.694 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:30.695 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:30.695 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:30.696 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:30.696 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:30.696 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:30.696 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.697 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:30.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:30.697 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:30.698 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:30.698 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:30.698 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:30.698 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:30.699 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:30.699 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:30.699 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:30.700 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.700 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:30.701 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:30.701 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:30.701 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:30.701 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:30.702 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:30.702 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:30.702 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:30.702 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:30.703 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:30.704 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:30.704 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:30.704 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:30.705 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:30.705 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:30.705 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:30.705 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:30.706 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:30.706 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.706 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:30.707 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:30.707 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:30.707 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:30.708 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:30.708 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:30.708 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:30.708 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:30.709 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:30.709 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:30.709 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.709 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.709 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:30.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:30.710 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:30.710 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:30.710 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:30.711 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:30.711 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:30.712 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:55:30.713 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:55:30.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:30.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:30.714 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:31.006 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:55:31.006 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:55:31.006 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:55:31.007 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:55:31.007 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:55:31.007 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:31.007 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:31.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:31.008 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:55:31.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:31.008 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:55:31.008 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:31.008 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:55:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:55:31.011 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:31.011 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:31.011 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:31.011 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:31.011 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:31.012 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:31.012 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:31.012 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:31.012 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:31.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:31.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:31.013 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:55:31.013 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:31.013 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:31.014 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:31.014 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:31.014 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:31.014 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:31.014 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:31.015 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:31.015 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:31.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:31.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:31.015 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:55:31.016 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:31.016 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:31.016 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:31.016 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:31.016 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:31.017 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:31.017 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:31.017 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:31.017 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:31.017 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:31.018 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:55:31.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:31.018 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:55:31.018 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:55:31.018 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:55:31.019 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:31.019 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:55:31.020 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:55:31.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:55:31.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:31.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:31.021 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:31.070 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:31.071 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:31.335 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:31.335 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:31.336 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:31.336 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:31.337 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:31.337 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:31.337 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:31.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:31.338 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:31.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:31.339 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:31.339 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:31.340 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:31.340 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:31.341 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:31.341 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:31.341 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:31.342 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:31.342 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:31.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:31.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:31.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:31.344 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:31.345 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:31.345 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:31.345 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:31.346 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:31.346 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:31.347 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:31.347 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:31.348 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:31.348 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:31.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:31.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:31.349 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:31.350 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:31.350 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:31.351 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:31.351 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:31.352 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:31.352 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:31.352 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:31.353 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:31.353 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:31.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:31.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:31.355 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:31.355 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:31.356 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:31.356 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:31.356 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:31.357 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:31.357 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:31.358 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:31.358 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:31.359 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:31.359 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:31.359 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:31.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:31.360 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:31.361 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:31.361 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:31.362 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:31.362 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:31.362 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:55:31.363 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-11-19T02:55:31.374 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-1 2020-11-19T02:55:35.070 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-2 2020-11-19T02:55:38.210 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-3 2020-11-19T02:55:38.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:38.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:38.288 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:38.324 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:38.325 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:38.532 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:55:38.564 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:55:38.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:38.564 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:38.564 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:53:34.640760Z" 2020-11-19T02:55:38.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:38.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:38.565 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754538.270230_85.1", 2020-11-19T02:55:38.565 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:38.270230Z" 2020-11-19T02:55:38.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:38.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:38.566 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:38.566 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:38.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:38.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:38.567 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:38.567 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:38.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:38.568 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:55:38.569 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:55:38.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:38.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:38.570 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:38.611 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:38.611 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:38.865 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:38.865 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:38.865 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:38.866 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:38.866 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:38.866 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:38.866 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:38.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:38.867 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:38.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:38.867 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:38.868 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:38.868 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:38.868 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:38.868 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:38.869 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:38.869 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:38.869 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:38.869 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:38.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:38.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:38.871 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:38.872 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:38.872 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:38.872 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:38.872 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754538.270230_85.1", 2020-11-19T02:55:38.873 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:38.873 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:38.873 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:38.873 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.270230Z", 2020-11-19T02:55:38.874 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:38.874 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:38.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:38.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:38.875 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:38.875 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:38.876 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:38.876 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:38.876 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:38.876 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:38.876 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:38.877 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:38.877 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:38.877 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:38.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:38.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:38.878 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:38.878 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:38.878 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:38.878 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:38.879 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:38.879 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:38.879 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:38.879 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:38.879 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:38.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:38.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:38.880 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:38.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:38.880 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:38.881 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:38.881 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:38.881 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:38.881 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:38.883 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:55:38.883 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:55:38.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:38.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:38.885 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:38.928 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:38.928 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:39.230 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:55:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:55:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:55:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:55:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:55:39.232 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:39.232 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:39.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:39.232 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:55:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:55:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:55:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:55:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754538.270230_85.1", 2020-11-19T02:55:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:38.270230Z", 2020-11-19T02:55:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:39.238 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:55:39.238 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:39.238 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:39.238 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:55:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:39.242 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:39.242 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:39.242 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:39.242 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:55:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:55:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:55:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:55:39.244 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:39.244 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:55:39.245 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:55:39.245 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:55:39.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:39.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:39.247 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:39.295 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:39.296 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:39.567 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:39.567 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:39.567 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:39.568 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:39.568 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:39.568 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:39.568 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:39.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:39.569 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:39.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:39.569 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:39.569 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:39.570 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:39.570 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:39.570 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:39.570 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:39.570 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:39.571 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:39.571 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:39.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:39.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:39.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:39.572 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:39.572 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:39.572 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:39.572 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754538.270230_85.1", 2020-11-19T02:55:39.572 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:39.573 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:39.573 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:39.573 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.270230Z", 2020-11-19T02:55:39.573 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:39.573 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:39.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:39.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:39.574 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:39.574 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:39.574 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:39.575 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:39.575 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:39.575 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:39.575 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:39.575 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:39.576 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:39.576 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:39.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:39.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:39.576 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:39.577 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:39.577 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:39.577 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:39.577 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:39.577 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:39.578 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:39.578 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:39.578 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:39.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:39.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:39.579 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:39.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:39.579 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:39.579 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:39.579 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:39.580 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:39.580 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:39.581 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:55:39.625 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-11-19T02:55:39.629 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-4 2020-11-19T02:55:39.662 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-5 2020-11-19T02:55:39.720 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-6 2020-11-19T02:55:39.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:39.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:39.767 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:39.801 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:39.802 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:40.020 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:55:40.052 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:55:40.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:40.053 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:40.053 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:53:34.640760Z" 2020-11-19T02:55:40.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:40.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:40.054 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754539.750854_98.1", 2020-11-19T02:55:40.055 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:39.750854Z" 2020-11-19T02:55:40.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:40.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:40.056 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:40.056 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:40.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:40.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:40.057 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:40.058 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:40.058 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:40.058 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:55:40.059 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:55:40.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:40.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:40.061 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:40.113 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:40.113 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:40.337 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:40.338 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:40.338 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:40.338 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:40.338 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:40.339 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:40.339 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.339 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:40.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:40.340 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:40.340 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:40.340 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:40.340 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:40.340 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:40.341 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:40.341 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:40.341 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:40.341 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.341 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:40.342 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:40.342 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:40.342 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:40.343 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754539.750854_98.1", 2020-11-19T02:55:40.343 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:40.343 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:40.343 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.750854Z", 2020-11-19T02:55:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:40.345 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:40.345 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:40.348 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:40.348 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:40.348 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:40.348 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:40.348 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:40.349 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:40.349 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:40.349 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:40.350 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.351 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.351 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.351 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:40.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.352 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:40.352 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:40.352 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:40.352 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.352 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:40.354 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:55:40.354 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:55:40.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:40.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:40.356 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:40.399 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:40.399 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:40.635 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:55:40.636 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:55:40.636 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:55:40.636 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:55:40.636 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:55:40.637 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:40.637 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:40.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:40.637 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:55:40.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:40.638 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:55:40.638 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:40.638 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:40.638 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:40.638 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:40.639 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:55:40.639 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:40.639 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:40.639 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:40.640 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:40.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:40.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:40.640 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:55:40.640 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:40.641 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:40.641 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754539.750854_98.1", 2020-11-19T02:55:40.641 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:40.641 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:40.641 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:40.642 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.750854Z", 2020-11-19T02:55:40.642 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:40.642 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:40.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:40.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:40.643 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:55:40.643 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:40.643 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:40.643 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:40.643 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:40.644 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:40.644 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:40.644 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:40.644 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:40.644 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:40.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:40.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:40.645 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:55:40.645 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:40.645 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:40.646 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:40.646 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:40.646 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:40.646 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:40.646 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:40.647 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:40.647 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:40.647 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:40.647 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:55:40.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:40.648 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:55:40.648 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:55:40.648 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:55:40.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:40.649 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:55:40.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:55:40.651 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:55:40.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:40.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:40.652 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:40.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:40.693 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:40.952 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:40.953 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:40.953 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:40.953 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:40.953 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:40.953 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:40.954 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.954 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:40.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:40.955 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:40.955 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:40.955 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:40.955 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:40.956 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:40.956 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:40.956 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:40.956 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:40.957 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:40.957 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:40.958 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:40.958 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:40.958 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754539.750854_98.1", 2020-11-19T02:55:40.958 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:40.959 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:40.959 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:40.959 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.750854Z", 2020-11-19T02:55:40.960 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.960 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:40.961 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:40.961 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:40.961 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:40.961 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:40.962 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:40.962 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:40.962 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:40.962 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:40.963 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.963 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:40.963 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:40.964 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:40.964 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:40.964 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:40.964 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:40.965 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:40.965 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:40.965 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:40.965 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:40.965 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.966 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.966 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:40.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:40.967 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:40.967 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:40.967 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:40.967 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:40.967 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:40.968 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:55:41.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:41.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:41.360 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:41.398 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:41.399 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:41.626 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:55:41.662 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:55:41.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:41.664 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:41.665 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:53:34.640760Z" 2020-11-19T02:55:41.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:41.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:41.667 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754539.750854_98.1", 2020-11-19T02:55:41.667 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:39.750854Z" 2020-11-19T02:55:41.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:41.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:41.670 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:41.671 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:41.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:41.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:41.672 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:41.673 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:41.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:41.673 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:55:41.674 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:55:41.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:41.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:41.676 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:41.717 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:41.718 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:41.965 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:41.966 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:41.966 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:41.966 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:41.966 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:41.967 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:41.967 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:41.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:41.967 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:41.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:41.968 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:41.968 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:41.968 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:41.968 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:41.968 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:41.969 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:41.969 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:41.969 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:41.969 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:41.970 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:41.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:41.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:41.970 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:41.970 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:41.971 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:41.971 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754539.750854_98.1", 2020-11-19T02:55:41.971 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:41.971 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:41.971 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:41.972 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.750854Z", 2020-11-19T02:55:41.972 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:41.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:41.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:41.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:41.973 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:41.973 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:41.973 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:41.973 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:41.973 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:41.974 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:41.974 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:41.974 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:41.974 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:41.974 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:41.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:41.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:41.975 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:41.975 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:41.975 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:41.976 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:41.976 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:41.976 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:41.976 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:41.976 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:41.977 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:41.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:41.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:41.977 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:41.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:41.978 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:41.978 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:41.978 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:41.978 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:41.978 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:41.980 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:55:41.980 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:55:41.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:41.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:41.981 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:42.033 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:42.033 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:42.273 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:55:42.274 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:55:42.275 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:55:42.275 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:55:42.276 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:55:42.277 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:42.279 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:42.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:42.279 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:55:42.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:42.280 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:55:42.281 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:42.281 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:42.281 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:42.282 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:42.282 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:55:42.283 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:42.283 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:42.283 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:42.284 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:42.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:42.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:42.285 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:55:42.285 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:42.285 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:42.285 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754539.750854_98.1", 2020-11-19T02:55:42.285 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:42.286 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:42.286 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:42.286 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.750854Z", 2020-11-19T02:55:42.286 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:42.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:42.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:42.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:42.287 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:55:42.287 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:42.288 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:42.288 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:42.288 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:42.288 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:42.288 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:42.289 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:42.289 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:42.289 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:42.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:42.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:42.290 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:55:42.290 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:42.291 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:42.291 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:42.291 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:42.291 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:42.292 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:42.292 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:42.292 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:42.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:42.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:42.293 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:55:42.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:42.293 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:55:42.293 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:55:42.293 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:55:42.294 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:42.294 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:55:42.294 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:55:42.295 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:55:42.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:42.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:42.296 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:42.334 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:42.335 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:42.604 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:42.604 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:42.604 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:42.605 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:42.605 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:42.605 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:42.605 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:42.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:42.606 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:42.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:42.606 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:42.606 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:42.607 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:42.607 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:42.607 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:42.607 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:42.607 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:42.608 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:42.608 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:42.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:42.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:42.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:42.609 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:42.609 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:42.609 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:42.609 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754539.750854_98.1", 2020-11-19T02:55:42.610 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:42.610 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:42.610 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:42.610 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.750854Z", 2020-11-19T02:55:42.610 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:42.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:42.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:42.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:42.611 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:42.611 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:42.612 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:42.612 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:42.612 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:42.612 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:42.613 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:42.613 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:42.613 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:42.613 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:42.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:42.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:42.614 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:42.614 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:42.614 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:42.614 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:42.615 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:42.615 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:42.615 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:42.615 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:42.616 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:42.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:42.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:42.616 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:42.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:42.617 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:42.617 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:42.617 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:42.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:42.617 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:42.618 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:55:42.662 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-11-19T02:55:42.663 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-7 2020-11-19T02:55:42.691 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-8 2020-11-19T02:55:42.750 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-9 2020-11-19T02:55:42.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:42.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:42.809 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:42.851 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:42.852 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:43.041 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:55:43.075 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:55:43.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:43.076 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:43.076 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:53:34.640760Z" 2020-11-19T02:55:43.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:43.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:43.077 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754542.787262_112.1", 2020-11-19T02:55:43.077 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:42.787262Z" 2020-11-19T02:55:43.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:43.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:43.077 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:43.078 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:43.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:43.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:43.078 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:43.079 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:43.079 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:43.079 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:55:43.080 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:55:43.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:43.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:43.082 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:43.121 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:43.122 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:43.367 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:43.368 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:43.368 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:43.368 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:43.368 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:43.369 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:43.369 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:43.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:43.369 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:43.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:43.370 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:43.370 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:43.371 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:43.371 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:43.371 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:43.371 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:43.372 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:43.372 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:43.372 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:43.372 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:43.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:43.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:43.373 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:43.373 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:43.373 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:43.374 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754542.787262_112.1", 2020-11-19T02:55:43.374 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:43.374 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:43.374 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:43.374 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.787262Z", 2020-11-19T02:55:43.375 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:43.375 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:43.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:43.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:43.375 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:43.376 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:43.376 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:43.376 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:43.376 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:43.376 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:43.377 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:43.377 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:43.377 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:43.377 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:43.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:43.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:43.378 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:43.378 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:43.378 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:43.379 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:43.379 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:43.379 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:43.379 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:43.380 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:43.380 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:43.380 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:43.380 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:43.380 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:43.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:43.381 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:43.381 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:43.382 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:43.382 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:43.382 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:43.383 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:55:43.384 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:55:43.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:43.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:43.386 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:43.426 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:43.426 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:43.660 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:55:43.660 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:55:43.661 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:55:43.661 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:55:43.661 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:55:43.661 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:43.662 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:43.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:43.662 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:55:43.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:43.662 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:55:43.663 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:43.663 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:43.663 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:43.663 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:43.664 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:55:43.664 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:43.664 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:43.664 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:43.664 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:43.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:43.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:43.665 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:55:43.665 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:43.665 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:43.666 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754542.787262_112.1", 2020-11-19T02:55:43.666 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:43.666 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:43.666 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:43.666 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.787262Z", 2020-11-19T02:55:43.667 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:43.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:43.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:43.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:43.667 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:55:43.668 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:43.668 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:43.668 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:43.668 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:43.668 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:43.669 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:43.669 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:43.669 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:43.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:43.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:43.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:43.670 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:55:43.670 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:43.670 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:43.671 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:43.671 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:43.671 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:43.671 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:43.671 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:43.672 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:43.672 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:43.672 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:43.672 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:55:43.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:43.673 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:55:43.673 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:55:43.673 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:55:43.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:43.673 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:55:43.674 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:55:43.675 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:55:43.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:43.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:43.677 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:43.717 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:43.717 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:43.995 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:43.995 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:43.995 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:43.996 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:43.996 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:43.997 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:43.997 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:43.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:43.998 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:43.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:43.999 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:44.000 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:44.000 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:44.001 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:44.001 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:44.004 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:44.005 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:44.005 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:44.005 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:44.005 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:44.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:44.006 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:44.006 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:44.006 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:44.007 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754542.787262_112.1", 2020-11-19T02:55:44.007 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:44.007 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:44.007 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:44.007 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.787262Z", 2020-11-19T02:55:44.008 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:44.008 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:44.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:44.008 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:44.009 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:44.009 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:44.009 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:44.009 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:44.009 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:44.010 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:44.010 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:44.010 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:44.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:44.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:44.011 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:44.011 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:44.011 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:44.011 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:44.012 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:44.012 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:44.012 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:44.012 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:44.012 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:44.014 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:44.014 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:44.014 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.014 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:44.015 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:55:44.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:44.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:44.334 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:44.373 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:44.374 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:44.598 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:55:44.629 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:55:44.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:44.630 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:44.630 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:53:34.640760Z" 2020-11-19T02:55:44.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:44.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:44.631 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:44.631 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T02:55:44.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:44.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:44.633 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:55:44.634 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:55:44.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:44.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:44.636 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:44.676 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:44.676 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:44.939 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:44.939 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:44.940 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:44.940 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:44.940 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:44.940 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:44.941 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:44.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:44.942 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:44.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:44.942 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:44.943 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:44.943 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:44.943 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:44.943 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:44.943 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:44.944 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:44.944 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:44.944 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:44.944 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:44.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:44.945 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:44.945 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:44.945 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:44.946 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:44.946 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:44.946 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:44.946 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:44.946 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:55:44.947 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:44.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:44.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:44.947 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:44.948 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:44.948 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:44.948 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:44.948 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:44.948 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:44.949 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:44.949 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:44.949 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:44.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:44.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:44.950 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:44.950 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:44.950 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:44.950 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:44.951 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:44.951 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:44.951 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:44.951 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:44.951 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:44.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.952 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:44.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:44.952 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:44.953 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:44.953 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:44.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:44.953 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:44.954 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:55:44.955 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:55:44.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:44.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:44.956 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:45.007 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:45.089 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:45.281 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:55:45.281 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:55:45.281 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:55:45.282 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:45.282 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:55:45.283 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:55:45.284 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:55:45.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:45.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:45.285 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:45.330 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:45.330 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:45.702 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:45.703 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:45.703 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:45.703 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:45.704 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:45.704 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:45.704 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:45.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:45.704 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:45.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:45.705 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:45.705 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:45.705 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:45.705 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754414.640760_109.1", 2020-11-19T02:55:45.706 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:45.706 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:45.706 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:45.706 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760Z", 2020-11-19T02:55:45.706 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:45.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:45.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:45.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:45.707 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:45.707 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:45.708 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:45.708 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:45.708 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:45.708 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:45.709 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:45.709 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:55:45.709 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:45.709 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:45.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:45.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:45.710 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:45.710 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:45.710 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:45.710 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:45.711 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:45.711 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:45.711 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:45.711 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:45.711 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:45.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:45.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:45.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:45.712 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:45.712 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:45.713 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:45.713 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:45.713 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:45.713 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:45.713 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:45.714 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:45.714 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:45.714 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:45.714 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:45.715 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:45.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:45.715 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:45.715 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:45.716 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:45.716 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:45.716 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:45.718 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:55:45.735 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-11-19T02:55:45.736 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-10 2020-11-19T02:55:45.766 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-11 2020-11-19T02:55:45.828 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-12 2020-11-19T02:55:46.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:46.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:46.611 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:46.662 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:46.662 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:55:46.926 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:55:46.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:46.926 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754545.866167_137.1", 2020-11-19T02:55:46.926 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:45.866167Z" 2020-11-19T02:55:46.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:46.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:46.927 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:46.927 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T02:55:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:46.930 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:55:46.931 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:55:46.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:46.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:46.932 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:46.985 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:46.986 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:47.203 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:47.203 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:47.204 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:47.204 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:47.204 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:47.204 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:47.205 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.205 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:47.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:47.205 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:47.206 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:47.206 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:47.206 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754545.866167_137.1", 2020-11-19T02:55:47.206 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:47.206 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:47.207 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:47.207 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.866167Z", 2020-11-19T02:55:47.207 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.207 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:47.208 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:47.208 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:47.208 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:47.209 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:47.209 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:47.209 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:47.209 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:47.210 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:55:47.210 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:47.211 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:47.211 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:47.211 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:47.211 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:47.211 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:47.212 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:47.212 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:47.212 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:47.212 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:47.213 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:47.213 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:47.213 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:47.214 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:47.214 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:47.214 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:47.214 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:47.214 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:47.215 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.215 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.215 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.215 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:47.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.216 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:47.216 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:47.216 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:47.216 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.216 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:47.218 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:55:47.218 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:55:47.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:47.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:47.219 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:47.263 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:47.263 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:47.522 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:55:47.522 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:55:47.522 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:55:47.523 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:55:47.523 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:55:47.523 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:47.523 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:47.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:47.524 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:55:47.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:47.524 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:55:47.524 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:47.525 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:47.525 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754545.866167_137.1", 2020-11-19T02:55:47.525 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:47.525 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:55:47.525 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:47.526 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.866167Z", 2020-11-19T02:55:47.526 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:47.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:47.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:47.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:47.527 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:55:47.527 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:47.527 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:47.527 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:47.527 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:47.528 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:47.528 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:47.528 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:55:47.528 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:47.528 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:47.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:47.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:47.529 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:55:47.529 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:47.529 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:47.530 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:47.530 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:47.530 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:47.530 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:47.531 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:47.531 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:47.531 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:47.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:47.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:47.532 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:55:47.532 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:47.532 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:47.532 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:47.533 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:47.535 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:47.536 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:47.536 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:47.536 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:47.536 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:47.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:47.537 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:55:47.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:47.537 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:55:47.537 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:55:47.538 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:55:47.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:47.538 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:55:47.539 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:55:47.539 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:55:47.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:47.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:47.540 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:47.581 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:47.582 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:47.848 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:47.849 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:47.849 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:47.849 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:47.849 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:47.850 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:47.850 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.851 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:47.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:47.851 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:47.851 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:47.851 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:47.852 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754545.866167_137.1", 2020-11-19T02:55:47.852 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:47.852 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:47.852 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:47.853 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.866167Z", 2020-11-19T02:55:47.853 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:47.854 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:47.854 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:47.855 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:47.855 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:47.855 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:47.855 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:47.856 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:47.856 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:55:47.856 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.856 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:47.857 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:47.857 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:47.857 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:47.857 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:47.858 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:47.858 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:47.858 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:47.858 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:47.859 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.859 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:47.860 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:47.860 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:47.860 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:47.861 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:47.861 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:47.861 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:47.861 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:47.862 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:47.862 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:47.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.863 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:47.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:47.863 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:47.864 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:47.864 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:47.864 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:47.864 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:47.866 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:55:47.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:47.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:47.868 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:55:47.954 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:47.954 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:48.197 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:55:48.197+0000 7fb3c61500c0 1 bucket sync caught up with source: 2020-11-19T02:55:48.198 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.4.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.4.5, , ] 2020-11-19T02:55:48.198 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.4.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.4.5, , ] 2020-11-19T02:55:48.198 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:55:48.197+0000 7fb3c61500c0 0 bucket checkpoint complete 2020-11-19T02:55:48.227 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-10 zones={test-zone2, test-zone1} 2020-11-19T02:55:48.239 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:55:48.248 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:48.248 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:48.249 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:48.249 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:48.250 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:55:48.258 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:48.258 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:48.259 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:48.259 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:48.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:48.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:48.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:48.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:48.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:48.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:48.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:48.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:49.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:49.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:49.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:49.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:49.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-10 zones={test-zone2, test-zone1} 2020-11-19T02:55:49.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:49.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:49.512 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:55:49.557 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:49.557 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:49.787 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:55:49.787+0000 7f4f1e7460c0 1 bucket sync caught up with source: 2020-11-19T02:55:49.788 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.4.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.4.5, , ] 2020-11-19T02:55:49.788 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.4.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.4.5, , ] 2020-11-19T02:55:49.788 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:55:49.787+0000 7f4f1e7460c0 0 bucket checkpoint complete 2020-11-19T02:55:49.810 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-10 zones={test-zone3, test-zone1} 2020-11-19T02:55:49.820 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:55:49.827 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:49.828 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:49.829 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:49.829 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:49.830 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:55:49.836 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:49.836 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:49.837 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:49.837 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:49.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:49.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:50.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:50.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:50.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:50.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:50.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:50.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:50.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:50.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:50.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:50.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:51.100 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-10 zones={test-zone3, test-zone1} 2020-11-19T02:55:51.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:51.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:51.102 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:55:51.156 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:51.156 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:51.411 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:55:51.409+0000 7f9bc9c6e0c0 1 bucket sync caught up with source: 2020-11-19T02:55:51.411 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.4.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.4.5, , ] 2020-11-19T02:55:51.412 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.4.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.4.5, , ] 2020-11-19T02:55:51.412 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:55:51.409+0000 7f9bc9c6e0c0 0 bucket checkpoint complete 2020-11-19T02:55:51.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:51.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:51.531 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:55:51.587 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:51.587 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:55:51.835+0000 7ff2db9950c0 1 bucket sync caught up with source: 2020-11-19T02:55:51.836 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.9.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.11.5, , ] 2020-11-19T02:55:51.836 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.9.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.11.5, , ] 2020-11-19T02:55:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:55:51.835+0000 7ff2db9950c0 0 bucket checkpoint complete 2020-11-19T02:55:51.863 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-11 zones={test-zone1, test-zone2} 2020-11-19T02:55:51.876 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:55:51.885 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:51.886 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:51.887 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:51.888 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:51.888 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:55:51.899 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:51.900 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:51.900 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:51.901 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:51.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:51.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:52.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:52.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:52.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:52.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:52.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:52.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:52.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:52.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:52.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:53.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:53.106 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-11 zones={test-zone1, test-zone2} 2020-11-19T02:55:53.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:53.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:53.108 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:55:53.151 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:53.152 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:53.379 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:55:53.379+0000 7f291fe670c0 1 bucket sync caught up with source: 2020-11-19T02:55:53.380 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.9.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.11.5, , ] 2020-11-19T02:55:53.380 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.9.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.11.5, , ] 2020-11-19T02:55:53.380 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:55:53.379+0000 7f291fe670c0 0 bucket checkpoint complete 2020-11-19T02:55:53.408 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-11 zones={test-zone3, test-zone2} 2020-11-19T02:55:53.418 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:55:53.425 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:53.425 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:53.425 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:53.426 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:53.426 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:55:53.433 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:53.434 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:53.434 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:53.434 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:53.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:53.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:53.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:53.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:53.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:53.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:53.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:54.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:54.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:54.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:54.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:54.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:54.614 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-11 zones={test-zone3, test-zone2} 2020-11-19T02:55:54.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:54.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:54.615 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:55:54.664 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:54.664 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:54.885 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:55:54.887+0000 7f459b2db0c0 1 bucket sync caught up with source: 2020-11-19T02:55:54.886 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.10.5, , 00000000001.9.5, , , , 00000000001.10.5, 00000000001.18.5, , ] 2020-11-19T02:55:54.886 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.10.5, , 00000000001.9.5, , , , 00000000001.10.5, 00000000001.18.5, , ] 2020-11-19T02:55:54.886 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:55:54.887+0000 7f459b2db0c0 0 bucket checkpoint complete 2020-11-19T02:55:54.914 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-12 zones={test-zone1, test-zone3} 2020-11-19T02:55:54.925 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:55:54.932 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:54.933 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:54.933 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:54.934 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:54.934 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:55:54.940 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:54.940 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:54.941 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:54.941 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:54.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:55.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:55.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:55.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:55.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:55.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:55.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:55.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:55.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:55.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:55.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:56.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:56.178 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-12 zones={test-zone1, test-zone3} 2020-11-19T02:55:56.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:56.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:56.180 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:55:56.239 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:56.239 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:56.440 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:55:56.441+0000 7fbb19e310c0 1 bucket sync caught up with source: 2020-11-19T02:55:56.441 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.10.5, , 00000000001.9.5, , , , 00000000001.10.5, 00000000001.18.5, , ] 2020-11-19T02:55:56.441 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.10.5, , 00000000001.9.5, , , , 00000000001.10.5, 00000000001.18.5, , ] 2020-11-19T02:55:56.441 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:55:56.441+0000 7fbb19e310c0 0 bucket checkpoint complete 2020-11-19T02:55:56.471 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-12 zones={test-zone2, test-zone3} 2020-11-19T02:55:56.482 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:55:56.490 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:56.491 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:56.491 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:56.491 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:56.492 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:55:56.501 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T02:55:56.501 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T02:55:56.502 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T02:55:56.502 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:56.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:56.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:56.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T02:55:56.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:56.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:57.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T02:55:57.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:57.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:57.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T02:55:57.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:57.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:57.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:57.769 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-12 zones={test-zone2, test-zone3} 2020-11-19T02:55:57.769 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-11-19T02:55:57.770 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-13 2020-11-19T02:55:57.797 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-14 2020-11-19T02:55:57.848 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-15 2020-11-19T02:55:58.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:58.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:58.068 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:58.109 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:58.110 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:58.289 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:55:58.324 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:55:58.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:58.325 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754557.888825_161.1", 2020-11-19T02:55:58.325 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:57.888825Z" 2020-11-19T02:55:58.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:58.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:58.326 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:58.327 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T02:55:58.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:58.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:58.328 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:58.328 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:58.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:58.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:58.329 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:58.329 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:55:58.329 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:58.330 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:55:58.331 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:55:58.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:58.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:58.332 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:58.370 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:58.371 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:58.594 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:58.595 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:58.595 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:58.595 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:58.595 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:58.595 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:58.596 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:58.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:58.596 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:58.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:58.596 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:58.597 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:58.597 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:58.597 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754557.888825_161.1", 2020-11-19T02:55:58.597 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:58.598 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:58.598 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:58.598 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.888825Z", 2020-11-19T02:55:58.598 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:58.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:58.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:58.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:58.599 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:58.599 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:58.599 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:58.600 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:58.600 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:58.600 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:58.600 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:58.600 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:55:58.600 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:58.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:58.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:58.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:58.601 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:58.602 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:58.602 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:58.602 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:58.602 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:58.602 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:58.603 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:58.603 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:58.603 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:58.603 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:58.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:58.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:58.604 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:58.604 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:58.604 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:58.604 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:58.605 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:58.605 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:58.605 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:58.605 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:58.605 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:58.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:58.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:58.606 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:58.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:58.607 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:58.608 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:58.608 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:58.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:58.609 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:58.610 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:55:58.611 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:55:58.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:58.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:58.614 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:58.653 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:58.653 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:58.882 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:55:58.883 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:55:58.883 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:55:58.884 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:55:58.884 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:55:58.884 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:58.885 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:58.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:58.885 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:55:58.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:58.886 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:55:58.887 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:58.887 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:58.887 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754557.888825_161.1", 2020-11-19T02:55:58.888 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:58.888 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:55:58.889 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:58.889 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.888825Z", 2020-11-19T02:55:58.889 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:58.890 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:58.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:58.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:58.891 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:55:58.891 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:58.892 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:58.892 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:58.892 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:58.893 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:58.893 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:58.894 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:55:58.894 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:58.894 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:58.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:58.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:58.896 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:55:58.896 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:58.896 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:58.897 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:58.897 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:58.898 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:58.898 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:58.898 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:58.899 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:58.899 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:58.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:58.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:55:58.900 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:55:58.901 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:55:58.901 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:55:58.901 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:55:58.902 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:55:58.902 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:55:58.903 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:55:58.903 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:55:58.903 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:55:58.904 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:58.904 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:58.904 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:55:58.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:55:58.905 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:55:58.906 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:55:58.906 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:55:58.906 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:55:58.907 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:55:58.909 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:55:58.909 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:55:58.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:58.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:55:58.912 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:55:58.953 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:58.953 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:59.320 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:55:59.321 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:55:59.321 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:55:59.321 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:55:59.321 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:55:59.322 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:55:59.322 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:59.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:59.322 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:55:59.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:59.323 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:55:59.323 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:59.323 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:59.323 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754557.888825_161.1", 2020-11-19T02:55:59.324 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:59.324 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:55:59.324 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:59.324 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.888825Z", 2020-11-19T02:55:59.325 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:59.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:59.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:59.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:59.325 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:55:59.326 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:59.326 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:59.326 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:55:59.326 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:59.326 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:55:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:59.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:59.328 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:55:59.328 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:59.328 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:59.328 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:59.329 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:59.329 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:59.329 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:59.329 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:59.329 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:59.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:59.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:59.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:55:59.330 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:55:59.330 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:55:59.331 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:55:59.331 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:55:59.331 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:55:59.331 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:55:59.331 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:55:59.332 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:55:59.332 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:55:59.332 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:59.332 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:59.332 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:55:59.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:55:59.333 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:55:59.333 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:55:59.334 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:55:59.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:55:59.334 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:55:59.336 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:55:59.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:59.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:55:59.339 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:55:59.433 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:55:59.434 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:55:59.652 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:55:59.653+0000 7f9f17b980c0 1 bucket sync caught up with source: 2020-11-19T02:55:59.653 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:55:59.653 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:55:59.653 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:55:59.653+0000 7f9f17b980c0 0 bucket checkpoint complete 2020-11-19T02:55:59.684 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-13 zones={test-zone2, test-zone1} 2020-11-19T02:55:59.696 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:55:59.746 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:59.747 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:55:59.754 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:55:59.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:55:59.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:00.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:00.147 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-13 zones={test-zone2, test-zone1} 2020-11-19T02:56:00.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:00.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:00.150 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:00.196 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:00.196 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:00.415+0000 7fb22008f0c0 1 bucket sync caught up with source: 2020-11-19T02:56:00.414 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:56:00.414 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:56:00.415 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:00.415+0000 7fb22008f0c0 0 bucket checkpoint complete 2020-11-19T02:56:00.441 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-13 zones={test-zone3, test-zone1} 2020-11-19T02:56:00.450 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:00.498 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:00.499 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:00.506 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:00.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:00.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:00.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:00.922 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-13 zones={test-zone3, test-zone1} 2020-11-19T02:56:00.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:00.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:00.925 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:00.975 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:00.976 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:01.210 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:01.209+0000 7fae282c60c0 1 bucket sync caught up with source: 2020-11-19T02:56:01.210 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:56:01.210 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:56:01.210 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:01.209+0000 7fae282c60c0 0 bucket checkpoint complete 2020-11-19T02:56:01.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:01.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:01.308 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:01.626 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:01.627+0000 7fb1a2a630c0 1 bucket sync caught up with source: 2020-11-19T02:56:01.626 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-19T02:56:01.627 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2020-11-19T02:56:01.628 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:01.627+0000 7fb1a2a630c0 0 bucket checkpoint complete 2020-11-19T02:56:01.656 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-14 zones={test-zone1, test-zone2} 2020-11-19T02:56:01.673 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:01.723 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:01.725 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:01.737 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:01.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:01.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:01.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:02.121 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-14 zones={test-zone1, test-zone2} 2020-11-19T02:56:02.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:02.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:02.125 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:02.174 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:02.391 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:02.391+0000 7f32a48920c0 1 bucket sync caught up with source: 2020-11-19T02:56:02.391 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-19T02:56:02.391 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2020-11-19T02:56:02.392 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:02.391+0000 7f32a48920c0 0 bucket checkpoint complete 2020-11-19T02:56:02.422 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-14 zones={test-zone3, test-zone2} 2020-11-19T02:56:02.434 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:02.483 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:02.484 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:02.495 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:02.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:02.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:02.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:02.843 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-14 zones={test-zone3, test-zone2} 2020-11-19T02:56:02.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:02.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:02.844 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:03.108 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:03.111+0000 7f84d122d0c0 1 bucket sync caught up with source: 2020-11-19T02:56:03.108 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.13.5, , , , , , , , , ] 2020-11-19T02:56:03.109 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.13.5, , , , , , , , , ] 2020-11-19T02:56:03.109 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:03.111+0000 7f84d122d0c0 0 bucket checkpoint complete 2020-11-19T02:56:03.137 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-15 zones={test-zone1, test-zone3} 2020-11-19T02:56:03.147 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:03.194 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:03.195 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:03.202 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:03.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:03.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:03.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:03.541 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-15 zones={test-zone1, test-zone3} 2020-11-19T02:56:03.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:03.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:03.543 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:03.590 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:03.849 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:03.849+0000 7f3dbf8620c0 1 bucket sync caught up with source: 2020-11-19T02:56:03.849 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.13.5, , , , , , , , , ] 2020-11-19T02:56:03.850 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.13.5, , , , , , , , , ] 2020-11-19T02:56:03.850 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:03.849+0000 7f3dbf8620c0 0 bucket checkpoint complete 2020-11-19T02:56:03.878 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-15 zones={test-zone2, test-zone3} 2020-11-19T02:56:03.890 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:03.938 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:03.939 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:03.946 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T02:56:04.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:04.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:04.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T02:56:04.316 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-15 zones={test-zone2, test-zone3} 2020-11-19T02:56:04.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:04.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:04.345 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:04.400 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:04.400 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:04.759 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:04.761+0000 7fb591f6b0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:04.760 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:56:04.760 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:09.764 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:09.765+0000 7fb591f6b0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:09.764 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:56:09.764 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:14.767 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:14.769+0000 7fb591f6b0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:14.768 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:56:14.768 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:19.773 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:19.773+0000 7fb591f6b0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:19.774 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-11-19T02:56:19.774 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:24.776 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:24.777+0000 7fb591f6b0c0 1 bucket sync caught up with source: 2020-11-19T02:56:24.776 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:24.776 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:24.776 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:24.777+0000 7fb591f6b0c0 0 bucket checkpoint complete 2020-11-19T02:56:24.806 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-13 zones={test-zone2, test-zone1} 2020-11-19T02:56:24.818 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:24.866 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:24.929 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-13 zones={test-zone2, test-zone1} 2020-11-19T02:56:24.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:24.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:24.931 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:24.977 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:24.977 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:25.196 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:25.199+0000 7f54e9d770c0 1 bucket sync caught up with source: 2020-11-19T02:56:25.196 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:25.196 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:25.197 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:25.199+0000 7f54e9d770c0 0 bucket checkpoint complete 2020-11-19T02:56:25.225 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-13 zones={test-zone3, test-zone1} 2020-11-19T02:56:25.237 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:25.289 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:25.631 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-13 zones={test-zone3, test-zone1} 2020-11-19T02:56:25.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:25.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:25.635 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:25.685 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:25.686 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:25.910 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:25.909+0000 7fb063a8c0c0 1 bucket sync caught up with source: 2020-11-19T02:56:25.911 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:25.911 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2020-11-19T02:56:25.911 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:25.909+0000 7fb063a8c0c0 0 bucket checkpoint complete 2020-11-19T02:56:25.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:25.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:25.967 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:26.011 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:26.011 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:26.223 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:26.223+0000 7f84a54510c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:26.224 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-19T02:56:26.224 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-19T02:56:31.226 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:31.227+0000 7f84a54510c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:31.227 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-19T02:56:31.227 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-19T02:56:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:36.231+0000 7f84a54510c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:36.230 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-19T02:56:36.230 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-19T02:56:41.234 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:41.235+0000 7f84a54510c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:41.234 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-19T02:56:41.234 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-19T02:56:46.237 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:46.239+0000 7f84a54510c0 1 bucket sync caught up with source: 2020-11-19T02:56:46.238 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2020-11-19T02:56:46.238 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-19T02:56:46.238 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:46.239+0000 7f84a54510c0 0 bucket checkpoint complete 2020-11-19T02:56:46.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-14 zones={test-zone1, test-zone2} 2020-11-19T02:56:46.276 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:46.328 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:46.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-14 zones={test-zone1, test-zone2} 2020-11-19T02:56:46.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:46.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:46.413 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:46.471 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:46.472 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:46.696 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:46.699+0000 7fadf47010c0 1 bucket sync caught up with source: 2020-11-19T02:56:46.696 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2020-11-19T02:56:46.696 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-19T02:56:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:46.699+0000 7fadf47010c0 0 bucket checkpoint complete 2020-11-19T02:56:46.727 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-14 zones={test-zone3, test-zone2} 2020-11-19T02:56:46.743 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:46.795 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:46.864 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-14 zones={test-zone3, test-zone2} 2020-11-19T02:56:46.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:46.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:46.889 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:46.942 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:46.942 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:47.154 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:47.155+0000 7f8ae503b0c0 1 bucket sync caught up with source: 2020-11-19T02:56:47.154 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000002.16.5, , , , , , , , , ] 2020-11-19T02:56:47.155 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000002.16.5, , , , , , , , , ] 2020-11-19T02:56:47.155 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:47.155+0000 7f8ae503b0c0 0 bucket checkpoint complete 2020-11-19T02:56:47.180 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-15 zones={test-zone1, test-zone3} 2020-11-19T02:56:47.192 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:47.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:47.306 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-15 zones={test-zone1, test-zone3} 2020-11-19T02:56:47.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:47.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:47.308 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:47.361 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:47.361 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:47.600 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:47.601+0000 7f2d31f520c0 1 bucket sync caught up with source: 2020-11-19T02:56:47.600 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.16.5, , , , , , , , , ] 2020-11-19T02:56:47.600 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.16.5, , , , , , , , , ] 2020-11-19T02:56:47.600 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:47.601+0000 7f2d31f520c0 0 bucket checkpoint complete 2020-11-19T02:56:47.630 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-15 zones={test-zone2, test-zone3} 2020-11-19T02:56:47.644 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:56:47.698 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:56:47.769 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-15 zones={test-zone2, test-zone3} 2020-11-19T02:56:47.771 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-11-19T02:56:47.774 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-16 2020-11-19T02:56:47.810 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-17 2020-11-19T02:56:47.883 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-18 2020-11-19T02:56:47.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:56:47.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:56:47.990 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:56:48.031 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:48.031 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:48.237 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:56:48.265 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:56:48.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:56:48.266 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754607.983290_227.1", 2020-11-19T02:56:48.266 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:56:47.983290Z" 2020-11-19T02:56:48.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:56:48.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:56:48.267 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:56:48.267 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T02:56:48.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:56:48.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:56:48.268 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:56:48.268 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:56:48.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:56:48.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:56:48.268 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:56:48.269 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:56:48.269 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:56:48.269 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:56:48.270 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:56:48.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:56:48.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:56:48.272 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:56:48.312 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:48.312 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:48.536 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:56:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:56:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:56:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:56:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:56:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:56:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:56:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:56:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:56:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:56:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:56:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754607.983290_227.1", 2020-11-19T02:56:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:56:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:56:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:56:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.983290Z", 2020-11-19T02:56:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:56:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:56:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:56:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:56:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:56:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:56:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:56:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:56:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:56:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:56:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:56:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:56:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:56:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:56:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:56:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:56:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:56:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:56:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:56:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:56:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:56:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:56:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:56:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:56:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:56:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:56:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:56:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:56:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:56:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:56:48.549 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:56:48.549 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:48.549 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:56:48.550 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:56:48.551 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:56:48.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:56:48.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:56:48.552 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:56:48.593 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:48.594 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:48.861 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:56:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:56:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:56:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:56:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:56:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:56:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:56:48.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:56:48.863 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:56:48.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:56:48.863 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:56:48.864 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:56:48.864 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:56:48.864 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754607.983290_227.1", 2020-11-19T02:56:48.864 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:56:48.864 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:56:48.865 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:56:48.865 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.983290Z", 2020-11-19T02:56:48.865 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:56:48.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:56:48.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:56:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:56:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:56:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:56:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:56:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:56:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:56:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:56:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:56:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:56:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:56:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:56:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:56:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:56:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:56:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:56:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:56:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:56:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:56:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:56:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:56:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:56:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:56:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:56:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:56:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:56:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:56:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:56:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:56:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:56:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:56:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:56:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:56:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:56:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:56:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:56:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:56:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:56:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:56:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:56:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:56:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:56:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:56:48.874 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:56:48.875 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:56:48.876 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:56:48.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:56:48.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:56:48.877 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:56:48.917 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:48.918 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:49.184 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:56:49.184 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:56:49.185 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:56:49.185 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:56:49.186 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:56:49.186 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:56:49.186 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:49.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:49.187 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:56:49.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:56:49.187 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:56:49.187 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:56:49.187 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:56:49.188 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754607.983290_227.1", 2020-11-19T02:56:49.188 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:56:49.188 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:56:49.188 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:56:49.188 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.983290Z", 2020-11-19T02:56:49.189 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:49.189 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:49.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:49.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:56:49.190 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:56:49.190 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:56:49.191 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:56:49.192 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:56:49.192 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:56:49.192 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:56:49.194 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:56:49.194 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:56:49.195 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:49.196 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:49.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:49.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:56:49.197 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:56:49.197 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:56:49.197 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:56:49.197 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:56:49.198 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:56:49.198 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:56:49.198 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:56:49.198 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:56:49.198 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:49.199 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:49.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:49.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:56:49.199 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:56:49.199 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:56:49.200 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:56:49.200 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:56:49.200 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:56:49.200 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:56:49.200 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:56:49.201 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:56:49.201 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:56:49.201 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:49.201 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:49.201 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:56:49.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:56:49.202 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:56:49.202 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:56:49.202 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:56:49.202 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:56:49.203 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:56:49.204 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:56:49.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:49.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:49.264 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:49.315 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:49.316 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:49.545 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:49.545+0000 7fbef38dd0c0 1 bucket sync caught up with source: 2020-11-19T02:56:49.546 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T02:56:49.546 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T02:56:49.546 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:56:49.545+0000 7fbef38dd0c0 0 bucket checkpoint complete 2020-11-19T02:56:49.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:49.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:56:49.577 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:56:49.629 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:56:49.629 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:56:49.850 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:49.851+0000 7f8d059dd0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:49.850 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T02:56:49.851 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:56:54.854 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:54.855+0000 7f8d059dd0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:54.854 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T02:56:54.855 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:56:59.858 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:56:59.859+0000 7f8d059dd0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:56:59.858 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T02:56:59.858 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:57:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:04.863+0000 7f8d059dd0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:04.862 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T02:57:04.862 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:57:09.865 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:09.866+0000 7f8d059dd0c0 1 bucket sync caught up with source: 2020-11-19T02:57:09.979 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:57:09.980 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:57:09.981 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:09.866+0000 7f8d059dd0c0 0 bucket checkpoint complete 2020-11-19T02:57:09.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:09.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:09.992 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:57:10.045 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:57:10.046 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:57:10.266 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:57:10.263+0000 7f56c4c6c0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:10.267 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:57:10.267 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T02:57:15.271 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:57:15.271+0000 7f56c4c6c0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:15.271 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:57:15.272 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T02:57:20.276 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:57:20.274+0000 7f56c4c6c0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:20.277 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:57:20.277 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T02:57:25.282 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:57:25.282+0000 7f56c4c6c0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:25.282 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T02:57:25.283 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T02:57:30.287 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:57:30.285+0000 7f56c4c6c0c0 1 bucket sync caught up with source: 2020-11-19T02:57:30.287 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T02:57:30.287 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T02:57:30.287 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:57:30.285+0000 7f56c4c6c0c0 0 bucket checkpoint complete 2020-11-19T02:57:30.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:30.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:30.387 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:57:30.436 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:57:30.438 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:57:30.648 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:30.647+0000 7f49ac4b00c0 1 bucket sync caught up with source: 2020-11-19T02:57:30.648 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.22.10, , , , , ] 2020-11-19T02:57:30.648 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000002.22.10, , , , , ] 2020-11-19T02:57:30.649 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:30.647+0000 7f49ac4b00c0 0 bucket checkpoint complete 2020-11-19T02:57:30.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:30.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:30.680 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:57:30.769 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:57:30.770 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:57:30.974 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:30.975+0000 7f9295b830c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:30.974 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.22.10, , , , , ] 2020-11-19T02:57:30.975 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000006.28.9, , , , , ] 2020-11-19T02:57:35.979 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:35.978+0000 7f9295b830c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:35.979 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.22.10, , , , , ] 2020-11-19T02:57:35.979 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000006.28.9, , , , , ] 2020-11-19T02:57:40.983 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:40.982+0000 7f9295b830c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:41.004 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.22.10, , , , , ] 2020-11-19T02:57:41.005 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000006.28.9, , , , , ] 2020-11-19T02:57:45.987 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:45.985+0000 7f9295b830c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:46.000 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.22.10, , , , , ] 2020-11-19T02:57:46.001 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000006.28.9, , , , , ] 2020-11-19T02:57:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:50.989+0000 7f9295b830c0 1 bucket sync caught up with source: 2020-11-19T02:57:50.992 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000006.28.9, , , , , ] 2020-11-19T02:57:50.992 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000006.28.9, , , , , ] 2020-11-19T02:57:50.992 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:50.989+0000 7f9295b830c0 0 bucket checkpoint complete 2020-11-19T02:57:51.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:51.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:51.085 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:57:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:57:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:57:51.348 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:51.345+0000 7f80a11350c0 1 bucket sync caught up with source: 2020-11-19T02:57:51.349 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T02:57:51.349 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T02:57:51.349 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:57:51.345+0000 7f80a11350c0 0 bucket checkpoint complete 2020-11-19T02:57:51.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:51.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:57:51.377 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:57:51.426 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:57:51.524 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:57:51.653 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:57:51.648+0000 7f25d69280c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:51.653 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T02:57:51.654 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T02:57:56.657 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:57:56.651+0000 7f25d69280c0 1 waiting for incremental sync to catch up: 2020-11-19T02:57:56.657 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T02:57:56.658 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T02:58:01.662 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:01.655+0000 7f25d69280c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:02.217 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T02:58:02.218 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T02:58:06.668 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:06.659+0000 7f25d69280c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:07.234 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T02:58:07.235 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T02:58:11.669 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:11.666+0000 7f25d69280c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:11.670 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T02:58:11.670 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T02:58:16.674 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:16.670+0000 7f25d69280c0 1 bucket sync caught up with source: 2020-11-19T02:58:16.675 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.18.9, , , , , ] 2020-11-19T02:58:16.676 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T02:58:16.677 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:16.670+0000 7f25d69280c0 0 bucket checkpoint complete 2020-11-19T02:58:18.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:18.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:18.515 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:18.566 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:18.566 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:58:18.762 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:18.758+0000 7f108b1510c0 1 bucket sync caught up with source: 2020-11-19T02:58:18.762 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.25.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:18.762 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.25.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:18.763 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:18.758+0000 7f108b1510c0 0 bucket checkpoint complete 2020-11-19T02:58:18.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:18.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:18.793 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:18.838 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:18.838 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:58:19.071 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:19.067+0000 7f3c2a0fc0c0 1 bucket sync caught up with source: 2020-11-19T02:58:19.071 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000010.25.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:19.071 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000010.25.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:19.072 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:19.067+0000 7f3c2a0fc0c0 0 bucket checkpoint complete 2020-11-19T02:58:19.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:19.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:19.097 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:19.145 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:19.146 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:58:19.379 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:19.374+0000 7f4b47c120c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:19.379 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.25.5, , 00000000002.9.10, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:19.380 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.25.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:24.384 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:24.378+0000 7f4b47c120c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:24.384 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.25.5, , 00000000002.9.10, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:24.384 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.25.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:29.389 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:29.385+0000 7f4b47c120c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:29.390 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.25.5, , 00000000002.9.10, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:29.390 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.25.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:34.394 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:34.389+0000 7f4b47c120c0 1 bucket sync caught up with source: 2020-11-19T02:58:34.394 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.25.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:34.395 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.25.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-11-19T02:58:34.395 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:34.389+0000 7f4b47c120c0 0 bucket checkpoint complete 2020-11-19T02:58:34.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:34.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:34.437 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:34.487 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:34.487 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:58:34.726 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:34.721+0000 7fcff9b4d0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:34.727 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.13.9, , 00000000010.30.5, , 00000000006.19.10, , , , , ] 2020-11-19T02:58:34.727 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.13.9, , 00000000010.30.5, , 00000000007.21.9, , , , , ] 2020-11-19T02:58:39.730 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:39.720+0000 7fcff9b4d0c0 1 bucket sync caught up with source: 2020-11-19T02:58:39.731 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.13.9, , 00000000010.30.5, , 00000000007.21.9, , , , , ] 2020-11-19T02:58:39.731 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.13.9, , 00000000010.30.5, , 00000000007.21.9, , , , , ] 2020-11-19T02:58:39.731 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:39.720+0000 7fcff9b4d0c0 0 bucket checkpoint complete 2020-11-19T02:58:39.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:39.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:39.762 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:39.808 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:39.809 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:58:40.041 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:40.032+0000 7f6cf27190c0 1 bucket sync caught up with source: 2020-11-19T02:58:40.041 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.13.9, , 00000000010.30.5, , 00000000007.21.9, , , , , ] 2020-11-19T02:58:40.042 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.13.9, , 00000000010.30.5, , 00000000007.21.9, , , , , ] 2020-11-19T02:58:40.042 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:40.032+0000 7f6cf27190c0 0 bucket checkpoint complete 2020-11-19T02:58:40.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:40.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:40.071 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:40.120 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:58:40.337 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:40.328+0000 7fb5c5d620c0 1 bucket sync caught up with source: 2020-11-19T02:58:40.337 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.16.9, , 00000000003.13.9, , 00000000014.37.5, , , , , ] 2020-11-19T02:58:40.337 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.16.9, , 00000000003.13.9, , 00000000014.37.5, , , , , ] 2020-11-19T02:58:40.337 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:40.328+0000 7fb5c5d620c0 0 bucket checkpoint complete 2020-11-19T02:58:40.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:40.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:40.365 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:40.426 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:40.427 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:58:40.650 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:40.645+0000 7f930e9e30c0 1 bucket sync caught up with source: 2020-11-19T02:58:40.650 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.16.9, , 00000000003.13.9, , 00000000014.37.5, , , , , ] 2020-11-19T02:58:40.650 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.16.9, , 00000000003.13.9, , 00000000014.37.5, , , , , ] 2020-11-19T02:58:40.651 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:40.645+0000 7f930e9e30c0 0 bucket checkpoint complete 2020-11-19T02:58:40.681 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-16 zones={test-zone2, test-zone1} 2020-11-19T02:58:40.694 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:58:40.701 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:58:40.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:58:40.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:58:40.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:58:40.703 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:58:40.712 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:58:40.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:58:40.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:58:40.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:58:40.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:58:40.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:58:40.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:58:40.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:58:40.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:58:41.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:58:41.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:58:41.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:58:41.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:58:41.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:58:41.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:58:41.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:58:41.965 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-16 zones={test-zone2, test-zone1} 2020-11-19T02:58:41.966 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-16 zones={test-zone3, test-zone1} 2020-11-19T02:58:41.981 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:58:41.996 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:58:41.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:58:41.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:58:41.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:58:41.998 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:58:42.012 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:58:42.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:58:42.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:58:42.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:58:42.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:58:42.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:58:42.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:58:42.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:58:42.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:58:42.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:58:42.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:58:42.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:58:42.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:58:42.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:58:42.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:58:43.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:58:43.132 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-16 zones={test-zone3, test-zone1} 2020-11-19T02:58:43.133 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-16 zones={test-zone3, test-zone2} 2020-11-19T02:58:43.145 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:58:43.152 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:58:43.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:58:43.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:58:43.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:58:43.154 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:58:43.162 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:58:43.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:58:43.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:58:43.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:58:43.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:58:43.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:58:43.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:58:43.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:58:43.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:58:43.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:58:43.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:58:43.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:58:43.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:58:43.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:58:43.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:58:44.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:58:44.158 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-16 zones={test-zone3, test-zone2} 2020-11-19T02:58:44.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:44.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:44.160 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:44.205 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:44.205 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:58:44.433 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:44.428+0000 7feb8e0200c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:44.433 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.20.5, , 00000000003.16.9, , 00000000004.21.9, , , , , ] 2020-11-19T02:58:44.433 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.20.5, , 00000000003.16.9, , 00000000007.27.9, , , , , ] 2020-11-19T02:58:49.438 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:49.432+0000 7feb8e0200c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:49.438 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.20.5, , 00000000003.16.9, , 00000000004.21.9, , , , , ] 2020-11-19T02:58:49.439 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.20.5, , 00000000003.16.9, , 00000000007.27.9, , , , , ] 2020-11-19T02:58:54.442 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:54.436+0000 7feb8e0200c0 1 waiting for incremental sync to catch up: 2020-11-19T02:58:54.442 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.20.5, , 00000000003.16.9, , 00000000004.21.9, , , , , ] 2020-11-19T02:58:54.443 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.20.5, , 00000000003.16.9, , 00000000007.27.9, , , , , ] 2020-11-19T02:58:59.446 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:59.440+0000 7feb8e0200c0 1 bucket sync caught up with source: 2020-11-19T02:58:59.447 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.20.5, , 00000000003.16.9, , 00000000007.27.9, , , , , ] 2020-11-19T02:58:59.447 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.20.5, , 00000000003.16.9, , 00000000007.27.9, , , , , ] 2020-11-19T02:58:59.448 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:58:59.440+0000 7feb8e0200c0 0 bucket checkpoint complete 2020-11-19T02:58:59.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:59.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:59.479 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:58:59.767 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:59.759+0000 7fee2f1a40c0 1 bucket sync caught up with source: 2020-11-19T02:58:59.767 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000010.20.5, , 00000000003.16.9, , 00000000007.27.9, , , , , ] 2020-11-19T02:58:59.768 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000010.20.5, , 00000000003.16.9, , 00000000007.27.9, , , , , ] 2020-11-19T02:58:59.768 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:58:59.759+0000 7fee2f1a40c0 0 bucket checkpoint complete 2020-11-19T02:58:59.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:59.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:58:59.796 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:58:59.846 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:58:59.847 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:00.082 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:00.076+0000 7fee212850c0 1 bucket sync caught up with source: 2020-11-19T02:59:00.082 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.20.5, , 00000000003.16.9, , 00000000007.27.9, , , , , ] 2020-11-19T02:59:00.082 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.20.5, , 00000000003.16.9, , 00000000007.27.9, , , , , ] 2020-11-19T02:59:00.082 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:00.076+0000 7fee212850c0 0 bucket checkpoint complete 2020-11-19T02:59:00.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:00.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:00.117 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:00.172 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:00.173 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:00.428 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:00.423+0000 7f82d2e970c0 1 bucket sync caught up with source: 2020-11-19T02:59:00.428 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.9.9, , 00000000010.41.5, , 00000000009.35.9, , , , , ] 2020-11-19T02:59:00.428 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.9.9, , 00000000010.41.5, , 00000000009.35.9, , , , , ] 2020-11-19T02:59:00.428 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:00.423+0000 7f82d2e970c0 0 bucket checkpoint complete 2020-11-19T02:59:00.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:00.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:00.461 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:00.506 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:00.507 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:00.742 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:00.735+0000 7f071b5a70c0 1 bucket sync caught up with source: 2020-11-19T02:59:00.743 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.9.9, , 00000000010.41.5, , 00000000009.35.9, , , , , ] 2020-11-19T02:59:00.744 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.9.9, , 00000000010.41.5, , 00000000009.35.9, , , , , ] 2020-11-19T02:59:00.744 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:00.735+0000 7f071b5a70c0 0 bucket checkpoint complete 2020-11-19T02:59:00.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:00.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:00.783 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:00.876 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:00.877 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:01.092 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:01.087+0000 7fa09acf10c0 1 bucket sync caught up with source: 2020-11-19T02:59:01.092 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.7.9, , 00000000003.21.9, , 00000000014.48.5, , , , , ] 2020-11-19T02:59:01.092 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.7.9, , 00000000003.21.9, , 00000000014.48.5, , , , , ] 2020-11-19T02:59:01.092 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:01.087+0000 7fa09acf10c0 0 bucket checkpoint complete 2020-11-19T02:59:01.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:01.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:01.119 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:01.170 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:01.170 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:01.398 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:01.392+0000 7f11f0fcc0c0 1 bucket sync caught up with source: 2020-11-19T02:59:01.398 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.7.9, , 00000000003.21.9, , 00000000014.48.5, , , , , ] 2020-11-19T02:59:01.399 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.7.9, , 00000000003.21.9, , 00000000014.48.5, , , , , ] 2020-11-19T02:59:01.399 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:01.392+0000 7f11f0fcc0c0 0 bucket checkpoint complete 2020-11-19T02:59:01.429 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-17 zones={test-zone2, test-zone1} 2020-11-19T02:59:01.439 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:01.446 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:01.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:01.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:01.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:01.448 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:01.470 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:01.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:01.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:01.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:01.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:01.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:01.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:01.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:01.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:01.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:02.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:02.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:02.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:02.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:02.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:02.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:02.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-17 zones={test-zone2, test-zone1} 2020-11-19T02:59:02.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-17 zones={test-zone3, test-zone1} 2020-11-19T02:59:02.857 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:02.865 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:02.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:02.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:02.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:02.868 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:02.876 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:02.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:02.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:02.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:02.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:02.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:02.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:03.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:03.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:03.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:03.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:03.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:03.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:03.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:03.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:03.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:03.913 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-17 zones={test-zone3, test-zone1} 2020-11-19T02:59:03.914 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-17 zones={test-zone3, test-zone2} 2020-11-19T02:59:03.924 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:03.931 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:03.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:03.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:03.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:03.932 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:03.940 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:03.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:03.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:03.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:03.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:03.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:04.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:04.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:04.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:04.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:04.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:04.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:04.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:04.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:04.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:04.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:04.948 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-17 zones={test-zone3, test-zone2} 2020-11-19T02:59:04.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:04.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:04.950 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:04.996 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:04.997 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:05.215 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:05.208+0000 7f529431b0c0 1 bucket sync caught up with source: 2020-11-19T02:59:05.216 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.22.5, , 00000000003.14.9, , 00000000007.16.9, , , , , ] 2020-11-19T02:59:05.216 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.22.5, , 00000000003.14.9, , 00000000007.16.9, , , , , ] 2020-11-19T02:59:05.216 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:05.208+0000 7f529431b0c0 0 bucket checkpoint complete 2020-11-19T02:59:05.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:05.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:05.242 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:05.291 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:05.292 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:05.519+0000 7f44cbd300c0 1 bucket sync caught up with source: 2020-11-19T02:59:05.524 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000010.22.5, , 00000000003.14.9, , 00000000007.16.9, , , , , ] 2020-11-19T02:59:05.524 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000010.22.5, , 00000000003.14.9, , 00000000007.16.9, , , , , ] 2020-11-19T02:59:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:05.519+0000 7f44cbd300c0 0 bucket checkpoint complete 2020-11-19T02:59:05.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:05.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:05.552 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:05.600 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:05.600 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:05.840 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:05.832+0000 7f37c54e80c0 1 bucket sync caught up with source: 2020-11-19T02:59:05.841 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.22.5, , 00000000003.14.9, , 00000000007.16.9, , , , , ] 2020-11-19T02:59:05.841 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.22.5, , 00000000003.14.9, , 00000000007.16.9, , , , , ] 2020-11-19T02:59:05.841 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:05.832+0000 7f37c54e80c0 0 bucket checkpoint complete 2020-11-19T02:59:05.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:05.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:05.969 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:06.251+0000 7feb723360c0 1 bucket sync caught up with source: 2020-11-19T02:59:06.258 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.12.9, , 00000000010.33.5, , 00000000007.19.9, , , , , ] 2020-11-19T02:59:06.258 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.12.9, , 00000000010.33.5, , 00000000007.19.9, , , , , ] 2020-11-19T02:59:06.259 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:06.251+0000 7feb723360c0 0 bucket checkpoint complete 2020-11-19T02:59:06.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:06.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:06.289 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:06.377 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:06.377 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:06.618 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:06.611+0000 7fa331cd90c0 1 bucket sync caught up with source: 2020-11-19T02:59:06.618 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.12.9, , 00000000010.33.5, , 00000000007.19.9, , , , , ] 2020-11-19T02:59:06.618 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.12.9, , 00000000010.33.5, , 00000000007.19.9, , , , , ] 2020-11-19T02:59:06.619 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:06.611+0000 7fa331cd90c0 0 bucket checkpoint complete 2020-11-19T02:59:06.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:06.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:06.651 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:06.706 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:06.707 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:06.926 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:06.919+0000 7fb4303520c0 1 bucket sync caught up with source: 2020-11-19T02:59:08.136 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000003.10.9, , 00000000003.16.9, , 00000000016.38.5, , , , , ] 2020-11-19T02:59:08.137 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000003.10.9, , 00000000003.16.9, , 00000000016.38.5, , , , , ] 2020-11-19T02:59:08.137 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:06.919+0000 7fb4303520c0 0 bucket checkpoint complete 2020-11-19T02:59:08.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:08.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:08.142 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:08.190 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:08.190 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:08.395 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:08.388+0000 7f8e0c0110c0 1 bucket sync caught up with source: 2020-11-19T02:59:08.396 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.10.9, , 00000000003.16.9, , 00000000016.38.5, , , , , ] 2020-11-19T02:59:08.396 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.10.9, , 00000000003.16.9, , 00000000016.38.5, , , , , ] 2020-11-19T02:59:08.397 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:08.388+0000 7f8e0c0110c0 0 bucket checkpoint complete 2020-11-19T02:59:08.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-18 zones={test-zone2, test-zone1} 2020-11-19T02:59:08.442 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:08.450 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:08.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:08.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:08.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:08.452 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:08.502 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:08.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:08.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:08.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:08.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:08.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:08.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:08.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:08.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:08.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:08.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:09.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:09.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:09.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:09.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:09.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:09.498 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-18 zones={test-zone2, test-zone1} 2020-11-19T02:59:09.499 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-18 zones={test-zone3, test-zone1} 2020-11-19T02:59:09.508 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:09.554 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:09.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:09.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:09.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:09.556 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:09.566 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:09.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:09.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:09.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:09.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:09.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:09.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:09.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:09.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:09.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:10.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:10.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:10.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:10.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:10.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:10.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:10.749 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-18 zones={test-zone3, test-zone1} 2020-11-19T02:59:10.750 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-18 zones={test-zone3, test-zone2} 2020-11-19T02:59:10.760 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:10.767 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:10.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:10.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:10.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:10.769 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:10.775 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:10.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:10.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:10.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:10.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:10.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:10.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:10.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:11.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:11.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:11.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:11.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:11.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:11.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:11.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:13.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:13.565 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-18 zones={test-zone3, test-zone2} 2020-11-19T02:59:14.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:14.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:14.207 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:14.258 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:14.258 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:14.470 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:14.464+0000 7f7e0016c0c0 1 waiting for incremental sync to catch up: 2020-11-19T02:59:14.471 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.25.5, , 00000000004.14.5, , 00000000010.25.5, , , , , ] 2020-11-19T02:59:14.471 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.31.5, , 00000000004.14.5, , 00000000010.25.5, , , , , ] 2020-11-19T02:59:19.475 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:19.468+0000 7f7e0016c0c0 1 bucket sync caught up with source: 2020-11-19T02:59:19.475 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.31.5, , 00000000004.14.5, , 00000000010.25.5, , , , , ] 2020-11-19T02:59:19.475 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.31.5, , 00000000004.14.5, , 00000000010.25.5, , , , , ] 2020-11-19T02:59:19.476 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:19.468+0000 7f7e0016c0c0 0 bucket checkpoint complete 2020-11-19T02:59:19.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:19.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:19.507 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:19.794+0000 7f157fb7c0c0 1 bucket sync caught up with source: 2020-11-19T02:59:19.803 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000011.31.5, , 00000000004.14.5, , 00000000010.25.5, , , , , ] 2020-11-19T02:59:19.803 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000011.31.5, , 00000000004.14.5, , 00000000010.25.5, , , , , ] 2020-11-19T02:59:19.804 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:19.794+0000 7f157fb7c0c0 0 bucket checkpoint complete 2020-11-19T02:59:19.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:19.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:19.828 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:19.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:19.878 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:20.108 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:20.100+0000 7f8469fdf0c0 1 bucket sync caught up with source: 2020-11-19T02:59:20.108 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.31.5, , 00000000004.14.5, , 00000000010.25.5, , , , , ] 2020-11-19T02:59:20.109 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.31.5, , 00000000004.14.5, , 00000000010.25.5, , , , , ] 2020-11-19T02:59:20.109 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:20.100+0000 7f8469fdf0c0 0 bucket checkpoint complete 2020-11-19T02:59:20.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:20.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:20.147 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:20.193 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:20.194 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:20.413 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:20.406+0000 7f2bd34690c0 1 bucket sync caught up with source: 2020-11-19T02:59:20.414 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000004.16.5, , 00000000011.37.5, , 00000000008.24.5, , , , , ] 2020-11-19T02:59:20.414 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000004.16.5, , 00000000011.37.5, , 00000000008.24.5, , , , , ] 2020-11-19T02:59:20.414 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:20.406+0000 7f2bd34690c0 0 bucket checkpoint complete 2020-11-19T02:59:20.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:20.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:20.446 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:20.492 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:20.492 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:20.716 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:20.710+0000 7f13f985d0c0 1 bucket sync caught up with source: 2020-11-19T02:59:20.716 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000004.16.5, , 00000000011.37.5, , 00000000008.24.5, , , , , ] 2020-11-19T02:59:20.717 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000004.16.5, , 00000000011.37.5, , 00000000008.24.5, , , , , ] 2020-11-19T02:59:20.717 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:20.710+0000 7f13f985d0c0 0 bucket checkpoint complete 2020-11-19T02:59:20.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:20.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:20.745 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:20.791 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:20.792 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:21.007 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:20.998+0000 7f159e8420c0 1 bucket sync caught up with source: 2020-11-19T02:59:21.008 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000005.23.5, , 00000000004.16.5, , 00000000015.40.5, , , , , ] 2020-11-19T02:59:21.008 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000005.23.5, , 00000000004.16.5, , 00000000015.40.5, , , , , ] 2020-11-19T02:59:21.009 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:20.998+0000 7f159e8420c0 0 bucket checkpoint complete 2020-11-19T02:59:21.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:21.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:21.037 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:21.098 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:21.099 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:21.308 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:21.300+0000 7feb6198f0c0 1 bucket sync caught up with source: 2020-11-19T02:59:21.308 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000005.23.5, , 00000000004.16.5, , 00000000015.40.5, , , , , ] 2020-11-19T02:59:21.308 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.23.5, , 00000000004.16.5, , 00000000015.40.5, , , , , ] 2020-11-19T02:59:21.309 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:21.300+0000 7feb6198f0c0 0 bucket checkpoint complete 2020-11-19T02:59:21.340 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-16 zones={test-zone2, test-zone1} 2020-11-19T02:59:21.351 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:21.359 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:21.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:21.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:21.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:21.361 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:21.366 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:21.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:21.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:21.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:21.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:21.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:21.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:21.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:21.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:21.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:23.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:23.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:23.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:24.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:24.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:24.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:24.301 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-16 zones={test-zone2, test-zone1} 2020-11-19T02:59:24.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-16 zones={test-zone3, test-zone1} 2020-11-19T02:59:24.310 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:24.315 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:24.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:24.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:24.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:24.317 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:24.322 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:24.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:24.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:24.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:24.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:24.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:24.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:24.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:24.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:24.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:24.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:24.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:24.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:25.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:25.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:25.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:25.373 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-16 zones={test-zone3, test-zone1} 2020-11-19T02:59:25.374 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-16 zones={test-zone3, test-zone2} 2020-11-19T02:59:25.384 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:25.390 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:25.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:25.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:25.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:25.392 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:25.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:25.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:25.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:25.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:25.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:25.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:25.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:25.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:25.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:26.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:26.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:26.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:26.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:26.486 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-16 zones={test-zone3, test-zone2} 2020-11-19T02:59:26.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:26.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:26.487 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:26.537 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:26.539 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:26.763 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:26.756+0000 7f4acf5380c0 1 bucket sync caught up with source: 2020-11-19T02:59:26.763 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.27.5, , 00000000004.19.5, , 00000000008.30.5, , , , , ] 2020-11-19T02:59:26.764 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.27.5, , 00000000004.19.5, , 00000000008.30.5, , , , , ] 2020-11-19T02:59:26.764 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:26.756+0000 7f4acf5380c0 0 bucket checkpoint complete 2020-11-19T02:59:26.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:29.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:29.202 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:29.249 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:29.475 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:29.466+0000 7f22556230c0 1 bucket sync caught up with source: 2020-11-19T02:59:29.475 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000011.27.5, , 00000000004.19.5, , 00000000008.30.5, , , , , ] 2020-11-19T02:59:29.476 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000011.27.5, , 00000000004.19.5, , 00000000008.30.5, , , , , ] 2020-11-19T02:59:29.476 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:29.466+0000 7f22556230c0 0 bucket checkpoint complete 2020-11-19T02:59:29.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:29.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:29.502 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:29.549 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:29.549 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:29.801 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:29.796+0000 7fad5945f0c0 1 bucket sync caught up with source: 2020-11-19T02:59:29.801 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.27.5, , 00000000004.19.5, , 00000000008.30.5, , , , , ] 2020-11-19T02:59:29.802 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.27.5, , 00000000004.19.5, , 00000000008.30.5, , , , , ] 2020-11-19T02:59:29.802 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:29.796+0000 7fad5945f0c0 0 bucket checkpoint complete 2020-11-19T02:59:29.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:29.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:29.917 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:30.178+0000 7f036c63a0c0 1 bucket sync caught up with source: 2020-11-19T02:59:30.185 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000004.12.5, , 00000000011.48.5, , 00000000010.38.5, , , , , ] 2020-11-19T02:59:30.186 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000004.12.5, , 00000000011.48.5, , 00000000010.38.5, , , , , ] 2020-11-19T02:59:30.187 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:30.178+0000 7f036c63a0c0 0 bucket checkpoint complete 2020-11-19T02:59:30.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:30.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:30.211 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:30.259 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:30.492 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:30.486+0000 7f816cc4a0c0 1 bucket sync caught up with source: 2020-11-19T02:59:30.492 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000004.12.5, , 00000000011.48.5, , 00000000010.38.5, , , , , ] 2020-11-19T02:59:30.492 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000004.12.5, , 00000000011.48.5, , 00000000010.38.5, , , , , ] 2020-11-19T02:59:30.493 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:30.486+0000 7f816cc4a0c0 0 bucket checkpoint complete 2020-11-19T02:59:30.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:30.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:30.520 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:30.611 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:30.611 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:30.830 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:30.822+0000 7f903b1b00c0 1 bucket sync caught up with source: 2020-11-19T02:59:30.830 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000004.10.5, , 00000000004.24.5, , 00000000015.51.5, , , , , ] 2020-11-19T02:59:30.830 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000004.10.5, , 00000000004.24.5, , 00000000015.51.5, , , , , ] 2020-11-19T02:59:30.830 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:30.822+0000 7f903b1b00c0 0 bucket checkpoint complete 2020-11-19T02:59:30.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:30.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:30.859 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:30.910 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:30.910 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:31.148 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:31.140+0000 7f77a43ac0c0 1 bucket sync caught up with source: 2020-11-19T02:59:31.149 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000004.10.5, , 00000000004.24.5, , 00000000015.51.5, , , , , ] 2020-11-19T02:59:31.149 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000004.10.5, , 00000000004.24.5, , 00000000015.51.5, , , , , ] 2020-11-19T02:59:31.149 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:31.140+0000 7f77a43ac0c0 0 bucket checkpoint complete 2020-11-19T02:59:31.181 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-17 zones={test-zone2, test-zone1} 2020-11-19T02:59:31.193 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:31.201 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:31.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:31.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:31.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:31.202 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:31.209 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:31.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:31.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:31.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:31.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:31.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:31.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:31.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:31.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:31.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:31.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:31.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:31.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:33.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:33.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:33.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:33.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-17 zones={test-zone2, test-zone1} 2020-11-19T02:59:33.715 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-17 zones={test-zone3, test-zone1} 2020-11-19T02:59:33.723 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:33.729 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:33.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:33.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:33.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:33.731 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:33.735 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:33.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:33.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:33.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:33.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:33.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:33.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:33.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:33.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:34.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:34.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:34.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:34.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:34.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:34.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:34.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:34.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-17 zones={test-zone3, test-zone1} 2020-11-19T02:59:34.764 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-17 zones={test-zone3, test-zone2} 2020-11-19T02:59:34.778 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:34.793 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:34.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:34.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:34.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:34.795 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:34.804 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:34.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:34.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:34.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:34.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:34.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:34.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:35.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:35.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:35.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:35.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:35.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:35.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:35.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:35.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:35.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:36.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-17 zones={test-zone3, test-zone2} 2020-11-19T02:59:36.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:36.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:36.002 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:36.049 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:36.049 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:36.287 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:36.280+0000 7f124f9130c0 1 bucket sync caught up with source: 2020-11-19T02:59:36.287 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.29.5, , 00000000004.17.5, , 00000000008.19.5, , , , , ] 2020-11-19T02:59:36.287 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.29.5, , 00000000004.17.5, , 00000000008.19.5, , , , , ] 2020-11-19T02:59:36.287 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:36.280+0000 7f124f9130c0 0 bucket checkpoint complete 2020-11-19T02:59:36.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:36.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:36.316 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:36.363 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:36.364 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:36.582 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:36.574+0000 7f8b238fe0c0 1 bucket sync caught up with source: 2020-11-19T02:59:36.584 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000011.29.5, , 00000000004.17.5, , 00000000008.19.5, , , , , ] 2020-11-19T02:59:36.584 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000011.29.5, , 00000000004.17.5, , 00000000008.19.5, , , , , ] 2020-11-19T02:59:36.586 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:36.574+0000 7f8b238fe0c0 0 bucket checkpoint complete 2020-11-19T02:59:36.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:36.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:36.614 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:36.663 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:36.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:36.896 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:36.888+0000 7f080a3990c0 1 bucket sync caught up with source: 2020-11-19T02:59:37.566 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.29.5, , 00000000004.17.5, , 00000000008.19.5, , , , , ] 2020-11-19T02:59:37.566 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.29.5, , 00000000004.17.5, , 00000000008.19.5, , , , , ] 2020-11-19T02:59:37.567 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:36.888+0000 7f080a3990c0 0 bucket checkpoint complete 2020-11-19T02:59:37.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:37.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:37.569 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:37.619 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:37.619 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:37.842+0000 7f6c34f980c0 1 bucket sync caught up with source: 2020-11-19T02:59:37.852 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000004.15.5, , 00000000011.40.5, , 00000000008.22.5, , , , , ] 2020-11-19T02:59:37.852 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000004.15.5, , 00000000011.40.5, , 00000000008.22.5, , , , , ] 2020-11-19T02:59:37.852 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:37.842+0000 7f6c34f980c0 0 bucket checkpoint complete 2020-11-19T02:59:37.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:37.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:37.883 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:37.969 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:37.970 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:38.208 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:38.202+0000 7f241b8550c0 1 bucket sync caught up with source: 2020-11-19T02:59:38.208 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000004.15.5, , 00000000011.40.5, , 00000000008.22.5, , , , , ] 2020-11-19T02:59:38.209 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000004.15.5, , 00000000011.40.5, , 00000000008.22.5, , , , , ] 2020-11-19T02:59:38.209 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:38.202+0000 7f241b8550c0 0 bucket checkpoint complete 2020-11-19T02:59:38.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:38.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:38.240 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:38.327 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:38.328 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:38.550+0000 7f7a094310c0 1 bucket sync caught up with source: 2020-11-19T02:59:38.558 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000004.13.5, , 00000000004.19.5, , 00000000017.41.5, , , , , ] 2020-11-19T02:59:38.558 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000004.13.5, , 00000000004.19.5, , 00000000017.41.5, , , , , ] 2020-11-19T02:59:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:38.550+0000 7f7a094310c0 0 bucket checkpoint complete 2020-11-19T02:59:38.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:38.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:38.586 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:38.634 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:38.635 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:38.878 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:38.872+0000 7f3e298870c0 1 bucket sync caught up with source: 2020-11-19T02:59:38.879 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000004.13.5, , 00000000004.19.5, , 00000000017.41.5, , , , , ] 2020-11-19T02:59:38.879 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000004.13.5, , 00000000004.19.5, , 00000000017.41.5, , , , , ] 2020-11-19T02:59:38.880 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:38.872+0000 7f3e298870c0 0 bucket checkpoint complete 2020-11-19T02:59:38.907 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-18 zones={test-zone2, test-zone1} 2020-11-19T02:59:38.917 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:38.924 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:38.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:38.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:38.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:38.926 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:38.931 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:38.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:38.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:38.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:38.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:39.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:39.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:39.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:39.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:39.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:39.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:39.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:39.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:39.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:39.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:39.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:40.071 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-18 zones={test-zone2, test-zone1} 2020-11-19T02:59:40.072 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-18 zones={test-zone3, test-zone1} 2020-11-19T02:59:40.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:40.096 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:40.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:40.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:40.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:40.098 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:40.105 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:40.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:40.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:40.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:40.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:40.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:40.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:40.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:40.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:40.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:40.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:40.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:40.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:40.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:40.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:41.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:41.243 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-18 zones={test-zone3, test-zone1} 2020-11-19T02:59:41.245 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-18 zones={test-zone3, test-zone2} 2020-11-19T02:59:41.258 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T02:59:41.264 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:41.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:41.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:41.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:41.266 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T02:59:41.271 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T02:59:41.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T02:59:41.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T02:59:41.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T02:59:41.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:41.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:41.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T02:59:41.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:41.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:41.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T02:59:41.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:41.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:41.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T02:59:43.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:43.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:43.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T02:59:43.838 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-18 zones={test-zone3, test-zone2} 2020-11-19T02:59:43.839 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-11-19T02:59:43.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:59:43.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:59:43.882 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:59:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:43.918 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:44.135 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T02:59:44.163 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T02:59:44.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:59:44.163 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754783.874924_266.1", 2020-11-19T02:59:44.164 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:43.874924Z" 2020-11-19T02:59:44.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:59:44.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:59:44.164 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:59:44.165 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T02:59:44.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:59:44.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:59:44.165 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:59:44.165 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:59:44.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:59:44.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:59:44.166 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:59:44.166 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T02:59:44.167 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:59:44.167 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T02:59:44.168 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T02:59:44.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:59:44.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:59:44.169 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:59:44.204 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:44.204 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:44.417 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:59:44.417 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:59:44.417 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:59:44.417 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:59:44.418 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:59:44.418 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:59:44.418 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:44.418 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:44.419 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:59:44.419 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:59:44.419 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:59:44.419 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:59:44.420 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:59:44.420 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754783.874924_266.1", 2020-11-19T02:59:44.420 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:59:44.420 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:59:44.420 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:59:44.421 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.874924Z", 2020-11-19T02:59:44.421 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:44.421 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:44.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:44.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:59:44.422 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:59:44.422 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:59:44.422 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:59:44.422 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:59:44.422 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:59:44.423 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:59:44.423 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:59:44.423 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:59:44.423 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:44.424 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:44.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:44.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:59:44.424 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:59:44.424 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:59:44.425 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:59:44.425 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:59:44.425 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:59:44.425 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:59:44.426 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:59:44.426 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:59:44.426 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:44.426 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:44.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:44.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:59:44.427 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:59:44.427 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:59:44.427 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:59:44.427 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:59:44.428 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:59:44.428 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:59:44.428 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:59:44.428 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:59:44.428 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:44.429 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:44.429 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:44.429 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:59:44.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:44.429 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:59:44.430 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:59:44.430 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:59:44.430 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:44.430 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:59:44.431 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T02:59:44.432 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T02:59:44.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:59:44.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:59:44.433 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:59:44.475 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:44.475 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:44.710 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T02:59:44.710 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T02:59:44.710 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T02:59:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T02:59:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T02:59:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:59:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:59:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:59:44.712 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T02:59:44.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:59:44.712 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T02:59:44.712 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:59:44.712 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:59:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754783.874924_266.1", 2020-11-19T02:59:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:59:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T02:59:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:59:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.874924Z", 2020-11-19T02:59:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:59:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:59:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:59:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:59:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T02:59:44.715 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:59:44.715 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:59:44.715 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:59:44.715 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:59:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:59:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:59:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:59:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:59:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:59:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:59:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:59:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T02:59:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:59:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:59:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:59:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:59:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:59:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:59:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:59:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:59:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:59:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:59:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T02:59:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T02:59:44.720 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T02:59:44.720 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T02:59:44.720 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T02:59:44.720 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T02:59:44.720 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T02:59:44.721 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T02:59:44.721 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T02:59:44.721 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T02:59:44.721 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:59:44.722 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:59:44.722 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T02:59:44.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T02:59:44.722 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T02:59:44.722 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T02:59:44.723 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T02:59:44.723 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T02:59:44.723 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T02:59:44.724 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T02:59:44.725 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T02:59:44.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:59:44.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T02:59:44.726 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T02:59:44.767 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:44.767 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:45.027 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T02:59:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T02:59:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T02:59:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T02:59:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T02:59:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T02:59:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T02:59:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:59:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T02:59:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:59:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:59:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754783.874924_266.1", 2020-11-19T02:59:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:59:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T02:59:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:59:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.874924Z", 2020-11-19T02:59:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:59:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T02:59:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:59:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:59:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T02:59:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:59:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:59:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:59:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T02:59:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:59:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T02:59:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:59:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:59:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:59:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:59:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:59:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:59:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:59:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T02:59:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T02:59:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T02:59:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T02:59:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T02:59:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T02:59:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T02:59:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T02:59:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T02:59:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T02:59:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T02:59:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T02:59:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T02:59:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T02:59:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T02:59:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T02:59:45.041 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T02:59:45.042 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T02:59:45.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:45.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:45.100 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:45.150 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:45.150 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:45.382 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:45.376+0000 7f55e65cd0c0 1 bucket sync caught up with source: 2020-11-19T02:59:45.382 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-11-19T02:59:45.382 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.9.10, , , , , ] 2020-11-19T02:59:45.382 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T02:59:45.376+0000 7f55e65cd0c0 0 bucket checkpoint complete 2020-11-19T02:59:45.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:45.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T02:59:45.413 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T02:59:45.467 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T02:59:45.467 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T02:59:45.690 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:45.682+0000 7f24159870c0 1 waiting for incremental sync to catch up: 2020-11-19T02:59:45.690 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-11-19T02:59:45.691 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-11-19T02:59:50.694 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:50.686+0000 7f24159870c0 1 waiting for incremental sync to catch up: 2020-11-19T02:59:50.694 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-11-19T02:59:50.694 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-11-19T02:59:55.704 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T02:59:55.690+0000 7f24159870c0 1 waiting for incremental sync to catch up: 2020-11-19T02:59:55.713 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-11-19T02:59:55.714 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-11-19T03:00:00.702 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:00.694+0000 7f24159870c0 1 waiting for incremental sync to catch up: 2020-11-19T03:00:00.791 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-11-19T03:00:00.792 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-11-19T03:00:05.705 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:05.698+0000 7f24159870c0 1 bucket sync caught up with source: 2020-11-19T03:00:05.706 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000005.15.9, , , , , ] 2020-11-19T03:00:05.706 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-11-19T03:00:05.706 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:05.698+0000 7f24159870c0 0 bucket checkpoint complete 2020-11-19T03:00:05.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:05.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:05.738 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:05.786 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:05.787 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:06.021 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:06.015+0000 7fbc754940c0 1 bucket sync caught up with source: 2020-11-19T03:00:06.021 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2020-11-19T03:00:06.021 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2020-11-19T03:00:06.022 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:06.015+0000 7fbc754940c0 0 bucket checkpoint complete 2020-11-19T03:00:06.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:06.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:06.055 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:06.108 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:06.243 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:06.351 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:06.346+0000 7f32fbc390c0 1 bucket sync caught up with source: 2020-11-19T03:00:06.352 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000004.18.9, , , , , ] 2020-11-19T03:00:06.352 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000004.18.9, , , , , ] 2020-11-19T03:00:06.352 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:06.346+0000 7f32fbc390c0 0 bucket checkpoint complete 2020-11-19T03:00:06.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:06.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:06.384 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:06.436 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:06.437 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:06.679 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:06.674+0000 7f38352740c0 1 bucket sync caught up with source: 2020-11-19T03:00:06.679 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000004.18.9, , , , , ] 2020-11-19T03:00:06.680 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000004.18.9, , , , , ] 2020-11-19T03:00:06.680 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:06.674+0000 7f38352740c0 0 bucket checkpoint complete 2020-11-19T03:00:06.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:06.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:06.710 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:07.043 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:07.034+0000 7f4d0aa1f0c0 1 bucket sync caught up with source: 2020-11-19T03:00:07.043 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:00:07.044 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:00:07.044 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:07.034+0000 7f4d0aa1f0c0 0 bucket checkpoint complete 2020-11-19T03:00:07.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:07.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:07.068 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:07.118 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:07.119 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:07.338 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:07.331+0000 7fe41979a0c0 1 bucket sync caught up with source: 2020-11-19T03:00:07.340 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:00:07.341 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:00:07.342 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:07.331+0000 7fe41979a0c0 0 bucket checkpoint complete 2020-11-19T03:00:07.370 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-19 zones={test-zone2, test-zone1} 2020-11-19T03:00:07.385 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:00:07.442 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:00:07.443 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:00:07.456 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:00:07.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:07.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:07.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:07.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-19 zones={test-zone2, test-zone1} 2020-11-19T03:00:07.715 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-19 zones={test-zone3, test-zone1} 2020-11-19T03:00:07.731 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:00:07.784 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:00:07.785 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:00:07.793 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:00:07.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:07.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:07.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:08.029 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-19 zones={test-zone3, test-zone1} 2020-11-19T03:00:08.030 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-19 zones={test-zone3, test-zone2} 2020-11-19T03:00:08.044 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:00:08.100 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:00:08.100 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:00:08.109 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:00:08.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:08.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:08.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:08.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-19 zones={test-zone3, test-zone2} 2020-11-19T03:00:09.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:09.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:09.966 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:10.018 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:10.018 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:10.254 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:10.247+0000 7fc7244780c0 1 waiting for incremental sync to catch up: 2020-11-19T03:00:10.254 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.15.9, , , 00000000010.27.5, , ] 2020-11-19T03:00:10.255 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000027.66.11, , ] 2020-11-19T03:00:15.260 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:15.251+0000 7fc7244780c0 1 waiting for incremental sync to catch up: 2020-11-19T03:00:15.261 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.15.9, , , 00000000010.27.5, , ] 2020-11-19T03:00:15.261 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000027.66.11, , ] 2020-11-19T03:00:20.264 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:20.255+0000 7fc7244780c0 1 waiting for incremental sync to catch up: 2020-11-19T03:00:20.350 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.15.9, , , 00000000010.27.5, , ] 2020-11-19T03:00:20.351 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000027.66.11, , ] 2020-11-19T03:00:25.267 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:25.259+0000 7fc7244780c0 1 waiting for incremental sync to catch up: 2020-11-19T03:00:25.268 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000010.27.5, , ] 2020-11-19T03:00:25.268 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000027.66.11, , ] 2020-11-19T03:00:30.271 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:30.263+0000 7fc7244780c0 1 bucket sync caught up with source: 2020-11-19T03:00:30.271 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000030.75.5, , ] 2020-11-19T03:00:30.272 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000027.66.11, , ] 2020-11-19T03:00:30.272 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:30.263+0000 7fc7244780c0 0 bucket checkpoint complete 2020-11-19T03:00:30.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:30.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:30.304 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:30.357 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:30.357 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:30.577+0000 7fa81e7b40c0 1 bucket sync caught up with source: 2020-11-19T03:00:30.583 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000044.113.9, , ] 2020-11-19T03:00:30.583 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000044.113.9, , ] 2020-11-19T03:00:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:30.577+0000 7fa81e7b40c0 0 bucket checkpoint complete 2020-11-19T03:00:30.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:30.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:30.616 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:30.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:30.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:30.919 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:30.911+0000 7fe82370c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:00:30.919 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000028.69.5, , ] 2020-11-19T03:00:30.919 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000044.113.9, , ] 2020-11-19T03:00:35.924 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:35.915+0000 7fe82370c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:00:36.035 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000028.69.5, , ] 2020-11-19T03:00:36.035 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000044.113.9, , ] 2020-11-19T03:00:40.927 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:40.919+0000 7fe82370c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:00:40.927 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000028.69.5, , ] 2020-11-19T03:00:40.927 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000044.113.9, , ] 2020-11-19T03:00:45.930 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:45.923+0000 7fe82370c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:00:45.931 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000028.69.5, , ] 2020-11-19T03:00:45.931 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000044.113.9, , ] 2020-11-19T03:00:50.934 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:50.927+0000 7fe82370c0c0 1 bucket sync caught up with source: 2020-11-19T03:00:50.935 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000045.115.9, , ] 2020-11-19T03:00:50.935 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000044.113.9, , ] 2020-11-19T03:00:50.935 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:50.927+0000 7fe82370c0c0 0 bucket checkpoint complete 2020-11-19T03:00:51.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:51.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:51.056 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:51.334 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:51.330+0000 7efee6f930c0 1 bucket sync caught up with source: 2020-11-19T03:00:51.334 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000047.118.9, , ] 2020-11-19T03:00:51.335 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000047.118.9, , ] 2020-11-19T03:00:51.335 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:51.330+0000 7efee6f930c0 0 bucket checkpoint complete 2020-11-19T03:00:51.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:51.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:51.367 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:51.458 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:51.458 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:51.695 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:51.686+0000 7f35f41fa0c0 1 bucket sync caught up with source: 2020-11-19T03:00:51.696 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000047.118.9, , ] 2020-11-19T03:00:51.697 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000047.118.9, , ] 2020-11-19T03:00:51.697 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:51.686+0000 7f35f41fa0c0 0 bucket checkpoint complete 2020-11-19T03:00:51.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:51.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:51.728 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:51.780 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:51.781 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:52.014 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:52.010+0000 7f5e720280c0 1 bucket sync caught up with source: 2020-11-19T03:00:52.015 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000050.125.9, , ] 2020-11-19T03:00:52.015 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000050.125.9, , ] 2020-11-19T03:00:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:00:52.010+0000 7f5e720280c0 0 bucket checkpoint complete 2020-11-19T03:00:52.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:52.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:00:52.044 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:00:52.094 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:00:52.095 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:00:52.346 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:52.339+0000 7f5b6fdec0c0 1 bucket sync caught up with source: 2020-11-19T03:00:52.346 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000050.125.9, , ] 2020-11-19T03:00:52.346 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000050.125.9, , ] 2020-11-19T03:00:52.346 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:00:52.339+0000 7f5b6fdec0c0 0 bucket checkpoint complete 2020-11-19T03:00:52.377 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-19 zones={test-zone2, test-zone1} 2020-11-19T03:00:52.396 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:00:52.412 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:00:52.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.428 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:00:52.446 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:00:52.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:00:52.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:52.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:52.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:00:52.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:52.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:52.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:53.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:54.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:54.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:54.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:54.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:54.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:54.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:54.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:54.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:54.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:55.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:55.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:55.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:55.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:55.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:55.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:55.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:55.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:56.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:56.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:56.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:56.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:56.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:56.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:56.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:56.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:57.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:57.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:57.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:57.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:57.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:57.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:57.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:57.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:58.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:59.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:59.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:59.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:59.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:59.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:59.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:59.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:59.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:00:59.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:00.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:01.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:01.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:01.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:01.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:01.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:01.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:01.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:01.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:02.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:02.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:02.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:02.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:02.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:03.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:03.154 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-19 zones={test-zone2, test-zone1} 2020-11-19T03:01:03.155 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-19 zones={test-zone3, test-zone1} 2020-11-19T03:01:03.164 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:01:03.171 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:01:03.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.184 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:01:03.196 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:01:03.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:03.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:01:03.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:01:03.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:01:03.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:03.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:03.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:03.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:03.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:03.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:03.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:03.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:04.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:04.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:04.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:04.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:04.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:04.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:04.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:04.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:04.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:05.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:06.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:06.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:06.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:06.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:06.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:06.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:06.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:06.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:06.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:08.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:09.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:10.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:10.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:10.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:10.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:10.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:10.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:10.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:10.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:10.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:11.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:13.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:13.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:13.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:13.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:13.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:13.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:13.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:13.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:13.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:14.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:14.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:14.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:14.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:14.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:14.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:14.642 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-19 zones={test-zone3, test-zone1} 2020-11-19T03:01:14.643 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-19 zones={test-zone3, test-zone2} 2020-11-19T03:01:14.665 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:01:14.682 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:01:14.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.707 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:01:14.720 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:01:14.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:14.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:01:14.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:01:14.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:01:14.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:15.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:16.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:16.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:16.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:16.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:16.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:16.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:16.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:16.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:18.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:19.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:20.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:20.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:20.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:20.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:20.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:20.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:20.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:20.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:20.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:21.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:21.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:21.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:21.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:21.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:21.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:21.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:21.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:21.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:23.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:23.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:24.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:25.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:25.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:25.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:25.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:25.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:25.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:25.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:25.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:25.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:26.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:27.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:27.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:27.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:27.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:28.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:28.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:28.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:28.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:28.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:28.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:28.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:28.755 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-19 zones={test-zone3, test-zone2} 2020-11-19T03:01:28.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:28.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:28.757 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:29.035 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:01:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:01:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:01:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:01:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:01:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:01:07.694734Z" 2020-11-19T03:01:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:29.039 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:01:29.040 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:01:29.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:29.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:29.042 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:29.084 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:29.085 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:29.321 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:01:29.322 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:01:29.322 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:01:29.322 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:01:29.322 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:01:29.323 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16121403893923359777 2020-11-19T03:01:29.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:29.323 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:01:29.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:29.323 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:01:29.324 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:29.324 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:29.324 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:01:29.324 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:29.324 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:29.325 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:29.325 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:01:29.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:29.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:29.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:29.326 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:01:29.326 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:29.326 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:29.326 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:29.327 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:29.327 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:29.327 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:29.327 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000" 2020-11-19T03:01:29.327 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:29.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:29.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:29.328 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:01:29.328 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:01:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:01:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:07.694734Z" 2020-11-19T03:01:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:01:29.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:29.333 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:01:29.333 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:01:29.333 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:01:29.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:29.334 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:01:29.335 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:01:29.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:29.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:29.336 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:29.375 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:29.376 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:29.616 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:01:29.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.617 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:01:29.617 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:01:29.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.618 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:29.618 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:01:29.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.619 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:29.619 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:01:29.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.620 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:29.620 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:01:07.694734Z" 2020-11-19T03:01:29.621 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:29.621 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:01:29.623 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:01:29.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:29.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:29.625 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:29.705 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:29.705 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:29.953 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:01:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:01:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:01:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:01:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:01:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7000157538207389965 2020-11-19T03:01:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:01:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:01:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:01:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:01:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:01:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000" 2020-11-19T03:01:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:01:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:01:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:01:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:07.694734Z" 2020-11-19T03:01:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:01:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:01:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:01:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:01:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:29.965 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:01:29.967 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:01:29.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:29.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:29.969 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:30.048 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:30.049 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:30.307 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:01:30.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:30.307 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:01:30.308 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:01:30.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:30.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:30.308 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:30.309 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:01:30.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:30.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:30.309 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:30.309 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:01:30.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:30.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:30.310 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:30.310 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:01:07.694734Z" 2020-11-19T03:01:30.311 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:30.311 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:01:30.312 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:01:30.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:30.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:30.313 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:30.365 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:30.366 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:30.630 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:01:30.630 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:01:30.630 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:01:30.631 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:01:30.631 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:01:30.631 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 12326036592617550149 2020-11-19T03:01:30.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:30.632 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:01:30.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:30.632 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:01:30.632 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:30.632 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:30.633 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:01:30.633 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:30.633 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:30.633 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:30.633 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:01:30.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:30.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:30.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:30.634 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:01:30.634 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:30.635 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:30.635 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:30.635 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:30.635 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:30.635 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:30.636 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000" 2020-11-19T03:01:30.636 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:30.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:30.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:30.636 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:01:30.637 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:30.637 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:30.637 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:30.637 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:30.637 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:30.638 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:30.638 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:01:30.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:30.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:30.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:30.639 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:01:30.639 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:30.639 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:30.639 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:30.640 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:30.640 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:30.640 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:30.640 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:07.694734Z" 2020-11-19T03:01:30.640 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:30.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:30.641 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:01:30.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:30.641 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:01:30.641 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:01:30.642 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:01:30.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:30.642 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:01:30.643 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:01:30.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:30.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:30.645 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:30.722 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:30.723 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:30.976 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:01:30.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:30.981 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:01:30.982 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:00:23.756845Z" 2020-11-19T03:01:30.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:30.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:30.987 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:30.988 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2020-11-19T03:01:30.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:30.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:30.991 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:30.992 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T02:59:14.134469Z" 2020-11-19T03:01:30.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:30.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:30.994 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:30.995 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:01:07.771688Z" 2020-11-19T03:01:30.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:30.996 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:01:30.999 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:01:31.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:31.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:31.001 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:31.046 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:31.047 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:31.281 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:01:31.282 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:01:31.282 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:01:31.282 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:01:31.283 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:01:31.283 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7306459910885960034 2020-11-19T03:01:31.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.284 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:01:31.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:31.285 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:01:31.285 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:31.286 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:31.286 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:01:31.286 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:31.286 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:31.287 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:31.287 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:00:23.756845Z" 2020-11-19T03:01:31.288 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:31.289 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:01:31.289 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:31.290 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:31.290 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:31.290 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:31.291 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:31.291 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:31.292 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000" 2020-11-19T03:01:31.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:31.293 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:01:31.294 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:31.295 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:31.295 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:31.295 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:31.295 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:31.296 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:31.296 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:14.134469Z" 2020-11-19T03:01:31.296 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:31.297 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:01:31.297 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:31.297 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:31.297 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:31.297 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:31.298 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:31.298 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:31.298 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:07.771688Z" 2020-11-19T03:01:31.298 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.298 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.299 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:01:31.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.299 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:01:31.299 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:01:31.299 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:01:31.300 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.300 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:01:31.301 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:01:31.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:31.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:31.302 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:31.341 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:31.342 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:31.603 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:01:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:01:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:00:23.756845Z" 2020-11-19T03:01:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2020-11-19T03:01:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T02:59:14.134469Z" 2020-11-19T03:01:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:01:07.771688Z" 2020-11-19T03:01:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:31.607 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:01:31.608 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:01:31.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:31.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:31.611 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:31.654 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:31.654 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:31.897 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:01:31.897 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:01:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:01:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:01:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:01:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 597164135494128594 2020-11-19T03:01:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:01:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:01:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:01:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:00:23.756845Z" 2020-11-19T03:01:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:01:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000" 2020-11-19T03:01:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:01:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:14.134469Z" 2020-11-19T03:01:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:01:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:01:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:07.771688Z" 2020-11-19T03:01:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:01:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:01:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:01:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:01:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:31.916 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:01:31.917 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:01:31.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:31.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:31.918 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:32.007 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:32.289 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:32.291 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:01:32.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:32.291 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:01:32.291 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.810439Z" 2020-11-19T03:01:32.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:32.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:32.292 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:32.292 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:01:32.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:32.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:32.293 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:32.293 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.283402Z" 2020-11-19T03:01:32.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:32.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:32.294 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-19T03:01:32.294 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:45.819785Z" 2020-11-19T03:01:32.294 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:32.294 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:01:32.295 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:01:32.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:32.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:32.296 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:32.378 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:32.638 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:01:32.639 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:01:32.641 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:01:32.642 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:01:32.642 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:01:32.642 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 2448524765768921960 2020-11-19T03:01:32.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:32.643 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:01:32.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:32.643 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:01:32.643 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:32.643 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:32.644 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:01:32.644 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:32.644 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:32.644 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:32.645 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:00:23.810439Z" 2020-11-19T03:01:32.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:32.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:32.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:32.645 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:01:32.646 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:32.646 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:32.646 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:32.646 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:32.647 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:32.647 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:32.647 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000" 2020-11-19T03:01:32.647 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:32.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:32.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:32.648 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:01:32.648 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:32.648 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:32.648 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:32.649 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:32.649 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:32.649 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:32.649 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:14.283402Z" 2020-11-19T03:01:32.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:32.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:32.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:32.650 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:01:32.650 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:32.650 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:01:32.651 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-19T03:01:32.651 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:32.651 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:32.651 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:32.651 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:00:45.819785Z" 2020-11-19T03:01:32.652 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:32.652 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:32.652 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:01:32.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:32.652 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:01:32.653 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:01:32.653 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:01:32.653 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:32.653 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:01:32.654 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:01:32.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:32.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:32.656 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:32.738 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:32.738 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:32.997 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:01:32.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:32.998 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:01:32.999 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.810439Z" 2020-11-19T03:01:32.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:33.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:33.000 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:33.001 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:01:33.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:33.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:33.003 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:33.004 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.283402Z" 2020-11-19T03:01:33.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:33.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:33.004 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-19T03:01:33.005 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:45.819785Z" 2020-11-19T03:01:33.005 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:33.005 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:01:33.006 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:01:33.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:33.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:33.008 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:33.055 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:33.055 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:33.297 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:01:33.298 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:01:33.299 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:01:33.299 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:01:33.299 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:01:33.299 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16483361435957904979 2020-11-19T03:01:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:01:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:01:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:33.301 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:33.301 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:01:33.301 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:33.301 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:00:23.810439Z" 2020-11-19T03:01:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:01:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:01:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000" 2020-11-19T03:01:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:01:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:01:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:14.283402Z" 2020-11-19T03:01:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:01:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:01:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-19T03:01:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:00:45.819785Z" 2020-11-19T03:01:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:01:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:01:33.310 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:01:33.310 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:01:33.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.310 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:01:33.311 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:01:33.312 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-11-19T03:01:33.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:33.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:33.353 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:33.391 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:33.391 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:33.600 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T03:01:33.632 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:01:33.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:33.633 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754893.346605_295.1", 2020-11-19T03:01:33.633 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:01:33.346605Z" 2020-11-19T03:01:33.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:33.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:33.634 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:01:33.634 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T03:01:33.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:33.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:33.634 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:01:33.635 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:01:33.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:33.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:33.635 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:01:33.635 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:01:33.636 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:33.636 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:01:33.637 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:01:33.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:33.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:33.638 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:33.679 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:33.679 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:33.931 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:01:33.931 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:01:33.931 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:01:33.932 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:01:33.932 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:01:33.932 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:01:33.932 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:33.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.933 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:01:33.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:33.933 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:01:33.933 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:33.933 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:33.934 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754893.346605_295.1", 2020-11-19T03:01:33.934 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:33.934 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:01:33.934 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:33.934 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.346605Z", 2020-11-19T03:01:33.935 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:33.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:33.935 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:01:33.936 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:33.936 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:33.936 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:01:33.936 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:33.937 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:33.937 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:33.937 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:01:33.937 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:33.937 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:33.938 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:01:33.938 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:33.938 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:33.939 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:01:33.939 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:33.939 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:33.939 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:33.939 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:01:33.940 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:33.940 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:33.940 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:01:33.941 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:33.941 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:33.941 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:01:33.941 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:33.941 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:33.942 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:33.942 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:01:33.942 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:33.942 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.942 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.943 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:01:33.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:33.943 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:01:33.943 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:01:33.943 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:01:33.944 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:33.944 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:01:33.945 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:01:33.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:01:33.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:33.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:33.947 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:33.987 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:33.987 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:34.223 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:01:34.223 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:01:34.224 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:01:34.224 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:01:34.224 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:01:34.224 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:01:34.224 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:34.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:34.225 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:01:34.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:34.225 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:01:34.226 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:34.226 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:01:34.226 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754893.346605_295.1", 2020-11-19T03:01:34.226 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:34.226 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:01:34.227 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:34.227 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:33.346605Z", 2020-11-19T03:01:34.227 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:34.227 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:34.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:34.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:34.228 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:01:34.228 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:34.228 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:01:34.229 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:01:34.229 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:34.229 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:34.229 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:34.230 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:01:34.230 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:34.230 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:34.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:34.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:34.232 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:01:34.232 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:34.233 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:01:34.233 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:01:34.233 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:34.233 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:34.234 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:34.234 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:01:34.234 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:34.234 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:34.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:34.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:34.235 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:01:34.235 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:34.235 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:01:34.236 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:01:34.236 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:34.236 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:34.236 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:34.237 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:01:34.237 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:34.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:34.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:34.237 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:01:34.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:34.238 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:01:34.238 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:01:34.238 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:01:34.238 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:34.239 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:01:34.240 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:01:34.240 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:01:34.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:34.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:34.242 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:34.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:34.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:34.541 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:01:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:01:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:01:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:01:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:01:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:01:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:01:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:01:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754893.346605_295.1", 2020-11-19T03:01:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:01:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.346605Z", 2020-11-19T03:01:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:01:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:01:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:01:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:01:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:01:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:01:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:01:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:01:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:01:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:01:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:01:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:01:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:01:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:34.555 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:01:34.556 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:01:34.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:34.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:34.607 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:34.656 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:34.657 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:34.878 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:34.871+0000 7f75b16080c0 1 bucket sync caught up with source: 2020-11-19T03:01:34.879 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.32.5, , ] 2020-11-19T03:01:34.879 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.32.5, , ] 2020-11-19T03:01:34.879 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:34.871+0000 7f75b16080c0 0 bucket checkpoint complete 2020-11-19T03:01:34.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:34.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:34.915 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:34.975 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:35.220 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:35.214+0000 7f558848b0c0 1 bucket sync caught up with source: 2020-11-19T03:01:35.220 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.32.5, , ] 2020-11-19T03:01:35.220 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.32.5, , ] 2020-11-19T03:01:35.220 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:35.214+0000 7f558848b0c0 0 bucket checkpoint complete 2020-11-19T03:01:35.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:35.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:35.248 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:35.300 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:35.300 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:35.527 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:35.523+0000 7f45bb1f20c0 1 bucket sync caught up with source: 2020-11-19T03:01:35.528 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.32.5, , ] 2020-11-19T03:01:35.528 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.32.5, , ] 2020-11-19T03:01:35.528 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:35.523+0000 7f45bb1f20c0 0 bucket checkpoint complete 2020-11-19T03:01:35.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:35.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:35.566 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:35.613 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:35.613 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:35.856 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:35.850+0000 7f8a394000c0 1 bucket sync caught up with source: 2020-11-19T03:01:35.857 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.18.5, , ] 2020-11-19T03:01:35.857 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.18.5, , ] 2020-11-19T03:01:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:35.850+0000 7f8a394000c0 0 bucket checkpoint complete 2020-11-19T03:01:35.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:35.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:35.888 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:35.937 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:35.938 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:36.171 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:36.166+0000 7fb1032910c0 1 bucket sync caught up with source: 2020-11-19T03:01:36.172 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.18.5, , ] 2020-11-19T03:01:36.172 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.18.5, , ] 2020-11-19T03:01:36.173 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:36.166+0000 7fb1032910c0 0 bucket checkpoint complete 2020-11-19T03:01:36.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:36.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:36.199 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:36.292 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:36.293 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:36.525 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:36.522+0000 7fc0eacba0c0 1 bucket sync caught up with source: 2020-11-19T03:01:36.526 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.16.5, , ] 2020-11-19T03:01:36.526 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.16.5, , ] 2020-11-19T03:01:36.526 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:36.522+0000 7fc0eacba0c0 0 bucket checkpoint complete 2020-11-19T03:01:36.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:36.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:36.555 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:36.605 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:36.606 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:36.852 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:36.851+0000 7fbb235a90c0 1 bucket sync caught up with source: 2020-11-19T03:01:36.852 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.16.5, , ] 2020-11-19T03:01:36.853 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.16.5, , ] 2020-11-19T03:01:36.853 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:36.851+0000 7fbb235a90c0 0 bucket checkpoint complete 2020-11-19T03:01:36.888 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-20 zones={test-zone2, test-zone1} 2020-11-19T03:01:36.908 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:01:38.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:38.176 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:01:38.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:38.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:38.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:38.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:38.412 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-20 zones={test-zone2, test-zone1} 2020-11-19T03:01:38.413 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-20 zones={test-zone3, test-zone1} 2020-11-19T03:01:38.423 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:01:38.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:38.473 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:01:38.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:38.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:38.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:38.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:38.680 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-20 zones={test-zone3, test-zone1} 2020-11-19T03:01:38.680 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-20 zones={test-zone3, test-zone2} 2020-11-19T03:01:38.691 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:01:38.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:38.739 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:01:38.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:01:38.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:38.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:38.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:01:38.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-20 zones={test-zone3, test-zone2} 2020-11-19T03:01:38.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:38.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:38.968 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:39.220 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T03:01:39.255 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:01:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754898.962533_304.1", 2020-11-19T03:01:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:01:38.962533Z" 2020-11-19T03:01:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:01:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T03:01:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:01:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:01:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:01:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:01:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:39.259 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:01:39.260 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:01:39.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:39.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:39.261 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:39.541 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:01:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:01:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:01:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:01:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:01:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:01:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:01:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:01:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754898.962533_304.1", 2020-11-19T03:01:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:01:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:38.962533Z", 2020-11-19T03:01:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:01:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:01:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:01:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:01:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:01:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:01:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:01:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:01:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:01:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:01:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:01:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:01:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:01:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:39.555 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:01:39.556 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:01:39.556 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:01:39.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:39.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:39.558 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:39.598 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:39.598 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:39.851 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:01:39.851 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:01:39.852 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:01:39.852 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:01:39.852 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:01:39.852 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:01:39.853 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:39.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:39.853 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:01:39.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:39.853 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:01:39.854 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:39.854 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:01:39.854 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754898.962533_304.1", 2020-11-19T03:01:39.854 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:39.855 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:01:39.855 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:39.855 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:38.962533Z", 2020-11-19T03:01:39.855 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:39.855 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:39.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:39.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:39.856 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:01:39.856 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:39.857 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:01:39.857 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:01:39.857 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:39.857 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:39.858 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:39.858 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:01:39.858 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:39.858 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:39.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:39.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:39.860 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:01:39.860 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:39.860 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:01:39.861 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:01:39.861 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:39.861 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:39.862 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:39.862 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:01:39.862 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:39.863 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:39.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:39.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:01:39.863 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:01:39.863 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:01:39.864 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:01:39.864 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:01:39.864 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:01:39.864 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:01:39.865 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:01:39.865 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:01:39.865 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:01:39.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:39.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:39.866 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:01:39.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:01:39.866 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:01:39.866 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:01:39.866 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:01:39.867 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:01:39.867 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:01:39.868 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:01:39.868 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:01:39.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:39.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:01:39.871 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:01:39.923 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:39.924 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:40.197 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:01:40.198 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:01:40.198 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:01:40.198 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:01:40.198 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:01:40.199 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:01:40.199 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:40.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:40.199 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:01:40.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:40.200 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:01:40.200 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:40.200 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:40.200 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754898.962533_304.1", 2020-11-19T03:01:40.200 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:40.201 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:01:40.201 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:40.201 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:38.962533Z", 2020-11-19T03:01:40.201 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:40.201 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:40.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:40.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:40.202 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:01:40.202 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:40.203 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:40.203 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:01:40.203 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:40.203 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:40.203 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:40.204 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:01:40.204 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:40.204 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:40.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:40.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:40.205 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:01:40.205 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:40.205 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:40.205 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:01:40.206 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:40.206 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:40.206 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:40.206 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:01:40.206 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:40.207 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:40.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:40.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:01:40.207 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:01:40.207 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:01:40.208 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:01:40.208 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:01:40.208 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:01:40.210 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:01:40.210 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:01:40.210 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:01:40.210 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:01:40.211 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:40.211 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:40.211 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:01:40.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:01:40.211 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:01:40.212 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:01:40.212 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:01:40.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:01:40.212 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:01:40.213 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:01:40.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:40.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:40.274 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:40.325 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:40.325 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:40.553 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:40.551+0000 7f404511e0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:01:40.553 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.32.5, , ] 2020-11-19T03:01:40.553 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2020-11-19T03:01:45.557 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:45.555+0000 7f404511e0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:01:45.558 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.32.5, , ] 2020-11-19T03:01:45.558 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2020-11-19T03:01:50.562 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:50.558+0000 7f404511e0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:01:50.562 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.32.5, , ] 2020-11-19T03:01:50.562 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2020-11-19T03:01:55.566 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:55.562+0000 7f404511e0c0 1 bucket sync caught up with source: 2020-11-19T03:01:55.566 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.35.12, , ] 2020-11-19T03:01:55.567 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2020-11-19T03:01:55.567 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:55.562+0000 7f404511e0c0 0 bucket checkpoint complete 2020-11-19T03:01:55.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:55.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:55.600 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:55.648 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:55.648 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:55.882 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:55.877+0000 7f448e70c0c0 1 bucket sync caught up with source: 2020-11-19T03:01:55.882 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.35.12, , ] 2020-11-19T03:01:55.882 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2020-11-19T03:01:55.882 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:55.877+0000 7f448e70c0c0 0 bucket checkpoint complete 2020-11-19T03:01:55.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:55.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:55.910 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:55.957 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:55.958 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:56.191 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:56.186+0000 7f45b5e550c0 1 bucket sync caught up with source: 2020-11-19T03:01:56.191 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.35.12, , ] 2020-11-19T03:01:56.191 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2020-11-19T03:01:56.192 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:01:56.186+0000 7f45b5e550c0 0 bucket checkpoint complete 2020-11-19T03:01:56.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:56.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:01:56.319 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:01:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:01:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:01:56.605 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:01:56.601+0000 7ff5d18880c0 1 waiting for incremental sync to catch up: 2020-11-19T03:01:56.605 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.18.5, , ] 2020-11-19T03:01:56.606 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.22.12, , ] 2020-11-19T03:02:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:01.605+0000 7ff5d18880c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:01.609 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.18.5, , ] 2020-11-19T03:02:01.610 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.22.12, , ] 2020-11-19T03:02:06.613 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:06.605+0000 7ff5d18880c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:06.613 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.18.5, , ] 2020-11-19T03:02:06.613 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.22.12, , ] 2020-11-19T03:02:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:11.609+0000 7ff5d18880c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:11.613 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.18.5, , ] 2020-11-19T03:02:11.613 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.22.12, , ] 2020-11-19T03:02:16.617 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:16.613+0000 7ff5d18880c0 1 bucket sync caught up with source: 2020-11-19T03:02:16.618 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000004.24.9, , ] 2020-11-19T03:02:16.618 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.22.12, , ] 2020-11-19T03:02:16.618 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:16.613+0000 7ff5d18880c0 0 bucket checkpoint complete 2020-11-19T03:02:16.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:16.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:16.650 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:02:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:16.969 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:16.965+0000 7f4a7dffd0c0 1 bucket sync caught up with source: 2020-11-19T03:02:17.053 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000004.24.9, , ] 2020-11-19T03:02:17.054 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000004.24.9, , ] 2020-11-19T03:02:17.054 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:16.965+0000 7f4a7dffd0c0 0 bucket checkpoint complete 2020-11-19T03:02:17.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:17.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:17.058 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:02:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:17.334 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:17.329+0000 7fade415d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:17.334 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2020-11-19T03:02:17.334 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:22.338 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:22.333+0000 7fade415d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:24.280 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2020-11-19T03:02:24.281 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:27.342 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:27.337+0000 7fade415d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:27.706 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2020-11-19T03:02:27.707 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:32.346 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:32.341+0000 7fade415d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:33.862 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2020-11-19T03:02:33.863 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:37.350 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:37.345+0000 7fade415d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:39.114 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2020-11-19T03:02:39.114 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:42.355 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:42.349+0000 7fade415d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:43.101 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2020-11-19T03:02:43.102 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:47.359 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:47.352+0000 7fade415d0c0 1 bucket sync caught up with source: 2020-11-19T03:02:47.919 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:47.919 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:47.919 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:47.352+0000 7fade415d0c0 0 bucket checkpoint complete 2020-11-19T03:02:47.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:47.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:47.922 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:02:47.972 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:47.973 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:48.207 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:02:48.201+0000 7fc864bce0c0 1 bucket sync caught up with source: 2020-11-19T03:02:48.207 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:48.208 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000004.22.9, , ] 2020-11-19T03:02:48.208 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:02:48.201+0000 7fc864bce0c0 0 bucket checkpoint complete 2020-11-19T03:02:48.235 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-20 zones={test-zone2, test-zone1} 2020-11-19T03:02:48.253 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:02:48.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:48.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:48.310 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:02:48.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:48.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:48.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:48.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:48.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:48.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:48.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:48.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:48.732 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-20 zones={test-zone2, test-zone1} 2020-11-19T03:02:48.733 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-20 zones={test-zone3, test-zone1} 2020-11-19T03:02:48.745 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:02:48.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:48.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:48.796 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:02:48.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:48.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:48.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:48.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:48.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:48.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:49.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:49.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:49.127 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-20 zones={test-zone3, test-zone1} 2020-11-19T03:02:49.128 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-20 zones={test-zone3, test-zone2} 2020-11-19T03:02:49.137 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:02:49.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:49.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:49.187 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:02:49.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:49.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:02:49.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:49.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:49.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:49.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:49.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:49.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:02:49.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-20 zones={test-zone3, test-zone2} 2020-11-19T03:02:49.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:02:49.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:02:49.556 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:02:49.594 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:49.594 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:49.814 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T03:02:49.847 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:02:49.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:02:49.848 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754969.549049_315.1", 2020-11-19T03:02:49.848 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:02:49.549049Z" 2020-11-19T03:02:49.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:02:49.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:02:49.849 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:02:49.849 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T03:02:49.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:02:49.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:02:49.850 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:02:49.850 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:02:49.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:02:49.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:02:49.851 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:02:49.851 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:02:49.851 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:02:49.851 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:02:49.852 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:02:49.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:02:49.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:02:49.854 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:02:49.894 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:49.894 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:50.145 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:02:50.146 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:02:50.146 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:02:50.146 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:02:50.146 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:02:50.147 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:02:50.147 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.147 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:02:50.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:02:50.148 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:02:50.148 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:02:50.148 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:02:50.148 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754969.549049_315.1", 2020-11-19T03:02:50.148 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:02:50.149 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:02:50.149 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:02:50.149 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:02:49.549049Z", 2020-11-19T03:02:50.149 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.149 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:02:50.150 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:02:50.150 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:02:50.150 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:02:50.151 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:02:50.151 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:02:50.151 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:02:50.151 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:02:50.151 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:02:50.152 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.152 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:02:50.152 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:02:50.153 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:02:50.153 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:02:50.153 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:02:50.153 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:02:50.153 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:02:50.154 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:02:50.154 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:02:50.154 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:02:50.155 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:02:50.155 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:02:50.155 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:02:50.156 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:02:50.156 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:02:50.156 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:02:50.156 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:02:50.156 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:02:50.157 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.157 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.157 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.157 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:02:50.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.158 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:02:50.158 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:02:50.158 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:02:50.158 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.158 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:02:50.160 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:02:50.160 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:02:50.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:02:50.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:02:50.162 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:02:50.202 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:50.202 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:50.446 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:02:50.446 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:02:50.446 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:02:50.447 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:02:50.447 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:02:50.447 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:02:50.447 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:02:50.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:02:50.448 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:02:50.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:02:50.448 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:02:50.448 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:02:50.449 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:02:50.449 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754969.549049_315.1", 2020-11-19T03:02:50.449 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:02:50.449 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:02:50.449 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:02:50.450 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:02:49.549049Z", 2020-11-19T03:02:50.450 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:02:50.450 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:02:50.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:02:50.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:02:50.451 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:02:50.451 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:02:50.451 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:02:50.451 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:02:50.451 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:02:50.452 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:02:50.452 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:02:50.452 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:02:50.452 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:02:50.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:02:50.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:02:50.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:02:50.453 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:02:50.453 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:02:50.454 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:02:50.454 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:02:50.454 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:02:50.454 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:02:50.454 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:02:50.455 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:02:50.455 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:02:50.455 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:02:50.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:02:50.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:02:50.456 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:02:50.456 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:02:50.456 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:02:50.456 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:02:50.456 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:02:50.457 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:02:50.457 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:02:50.457 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:02:50.457 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:02:50.457 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:02:50.458 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:02:50.458 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:02:50.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:02:50.458 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:02:50.458 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:02:50.459 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:02:50.459 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:02:50.459 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:02:50.460 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:02:50.461 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:02:50.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:02:50.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:02:50.462 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:02:50.506 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:50.506 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:50.778 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:02:50.778 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:02:50.778 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:02:50.779 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:02:50.779 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:02:50.779 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:02:50.780 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.780 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:02:50.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:02:50.781 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:02:50.781 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:02:50.781 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:02:50.781 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754969.549049_315.1", 2020-11-19T03:02:50.783 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:02:50.784 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:02:50.786 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:02:50.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:02:49.549049Z", 2020-11-19T03:02:50.787 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:02:50.788 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:02:50.788 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:02:50.789 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:02:50.789 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:02:50.789 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:02:50.789 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:02:50.789 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:02:50.790 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:02:50.790 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:02:50.791 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:02:50.791 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:02:50.791 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:02:50.791 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:02:50.792 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:02:50.792 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:02:50.792 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:02:50.792 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:02:50.792 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:02:50.793 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:02:50.793 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:02:50.794 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:02:50.794 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:02:50.794 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:02:50.794 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:02:50.794 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:02:50.795 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:02:50.795 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:02:50.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.796 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:02:50.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:02:50.797 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:02:50.797 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:02:50.797 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:02:50.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:02:50.798 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:02:50.799 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:02:50.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:50.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:50.856 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:02:50.908 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:50.908 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:51.122 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:02:51.117+0000 7fc110f3d0c0 1 bucket sync caught up with source: 2020-11-19T03:02:51.122 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000005.39.17, , ] 2020-11-19T03:02:51.122 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.39.17, , ] 2020-11-19T03:02:51.123 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:02:51.117+0000 7fc110f3d0c0 0 bucket checkpoint complete 2020-11-19T03:02:51.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:51.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:51.149 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:02:51.194 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:51.194 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:51.419 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:51.412+0000 7fdcad83d0c0 1 bucket sync caught up with source: 2020-11-19T03:02:51.420 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000005.39.17, , ] 2020-11-19T03:02:51.420 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000005.39.17, , ] 2020-11-19T03:02:51.420 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:51.412+0000 7fdcad83d0c0 0 bucket checkpoint complete 2020-11-19T03:02:51.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:51.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:51.448 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:02:51.497 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:51.497 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:51.727 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:02:51.721+0000 7f7810f4a0c0 1 bucket sync caught up with source: 2020-11-19T03:02:51.729 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000005.39.17, , ] 2020-11-19T03:02:51.729 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.39.17, , ] 2020-11-19T03:02:51.729 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:02:51.721+0000 7f7810f4a0c0 0 bucket checkpoint complete 2020-11-19T03:02:51.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:51.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:02:51.763 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:02:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:02:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:02:52.044 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:52.036+0000 7ff0180c20c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:52.045 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000004.24.9, , ] 2020-11-19T03:02:52.045 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000006.29.17, , ] 2020-11-19T03:02:57.049 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:02:57.040+0000 7ff0180c20c0 1 waiting for incremental sync to catch up: 2020-11-19T03:02:57.050 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000004.24.9, , ] 2020-11-19T03:02:57.051 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000006.29.17, , ] 2020-11-19T03:03:02.052 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:02.044+0000 7ff0180c20c0 1 waiting for incremental sync to catch up: 2020-11-19T03:03:02.053 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000004.24.9, , ] 2020-11-19T03:03:02.053 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000006.29.17, , ] 2020-11-19T03:03:07.056 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:07.048+0000 7ff0180c20c0 1 bucket sync caught up with source: 2020-11-19T03:03:07.057 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000007.31.14, , ] 2020-11-19T03:03:07.057 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000006.29.17, , ] 2020-11-19T03:03:07.057 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:07.048+0000 7ff0180c20c0 0 bucket checkpoint complete 2020-11-19T03:03:07.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:07.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:07.079 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:07.128 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:07.129 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:07.356 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:07.348+0000 7f70694560c0 1 bucket sync caught up with source: 2020-11-19T03:03:07.357 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000007.31.14, , ] 2020-11-19T03:03:07.357 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000007.31.14, , ] 2020-11-19T03:03:07.357 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:07.348+0000 7f70694560c0 0 bucket checkpoint complete 2020-11-19T03:03:07.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:07.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:07.389 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:07.480 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:07.481 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:07.696+0000 7f7dbfcbc0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:03:07.704 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000006.27.17, , ] 2020-11-19T03:03:07.705 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000007.29.14, , ] 2020-11-19T03:03:12.709 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:12.704+0000 7f7dbfcbc0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:03:12.709 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000006.27.17, , ] 2020-11-19T03:03:12.709 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000007.29.14, , ] 2020-11-19T03:03:17.714 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:17.712+0000 7f7dbfcbc0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:03:18.375 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000006.27.17, , ] 2020-11-19T03:03:18.375 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000007.29.14, , ] 2020-11-19T03:03:22.718 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:22.716+0000 7f7dbfcbc0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:03:23.747 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000006.27.17, , ] 2020-11-19T03:03:23.747 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000007.29.14, , ] 2020-11-19T03:03:27.721 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:27.720+0000 7f7dbfcbc0c0 1 bucket sync caught up with source: 2020-11-19T03:03:27.722 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000007.29.14, , ] 2020-11-19T03:03:27.722 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000007.29.14, , ] 2020-11-19T03:03:27.723 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:27.720+0000 7f7dbfcbc0c0 0 bucket checkpoint complete 2020-11-19T03:03:27.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:27.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:27.751 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:27.809 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:27.810 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:28.061 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:28.057+0000 7f08b0e950c0 1 bucket sync caught up with source: 2020-11-19T03:03:28.062 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000007.29.14, , ] 2020-11-19T03:03:28.062 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000007.29.14, , ] 2020-11-19T03:03:28.062 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:28.057+0000 7f08b0e950c0 0 bucket checkpoint complete 2020-11-19T03:03:28.090 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-20 zones={test-zone2, test-zone1} 2020-11-19T03:03:28.104 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:28.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:28.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:28.156 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:28.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:28.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:28.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.584 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-20 zones={test-zone2, test-zone1} 2020-11-19T03:03:28.584 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-20 zones={test-zone3, test-zone1} 2020-11-19T03:03:28.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:28.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:28.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:28.650 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:28.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:28.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:28.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:28.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:29.041 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-20 zones={test-zone3, test-zone1} 2020-11-19T03:03:29.042 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-20 zones={test-zone3, test-zone2} 2020-11-19T03:03:29.053 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:29.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:29.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:29.103 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:29.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:29.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:29.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:29.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:29.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:29.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:29.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:29.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:29.460 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-20 zones={test-zone3, test-zone2} 2020-11-19T03:03:29.461 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-11-19T03:03:29.462 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-21 2020-11-19T03:03:29.497 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-22 2020-11-19T03:03:29.553 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-23 2020-11-19T03:03:29.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:29.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:29.687 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:29.723 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:29.724 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:29.917 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T03:03:29.945 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:03:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755009.660851_356.1", 2020-11-19T03:03:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:03:29.660851Z" 2020-11-19T03:03:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T03:03:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:03:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:03:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:29.949 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:03:29.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:03:29.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:29.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:29.951 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:29.991 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:30.237 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:03:30.238 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:03:30.238 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:03:30.239 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:03:30.239 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:03:30.239 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:03:30.240 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.241 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:03:30.241 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:30.241 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:03:30.242 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:30.242 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:30.243 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755009.660851_356.1", 2020-11-19T03:03:30.243 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:30.243 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:03:30.244 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:30.244 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.660851Z", 2020-11-19T03:03:30.244 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.244 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:30.246 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:03:30.246 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:30.246 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:30.247 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:30.247 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:30.248 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:30.248 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:30.249 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:03:30.249 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:30.250 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:03:30.251 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:30.251 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:30.251 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:30.252 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:30.252 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:30.252 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:30.253 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:30.253 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:30.254 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:03:30.254 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:30.255 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:30.255 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:30.255 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:30.256 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:30.256 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:30.256 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:30.257 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.257 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.257 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.258 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:03:30.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.259 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:03:30.259 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:03:30.259 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:03:30.259 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.260 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:03:30.261 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:03:30.262 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:03:30.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:30.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:30.264 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:30.310 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:30.311 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:30.568 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:03:30.568 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:03:30.569 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:03:30.569 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:03:30.569 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:03:30.570 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:03:30.570 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:30.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:30.570 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:03:30.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:30.572 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:03:30.572 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:30.572 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:30.573 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755009.660851_356.1", 2020-11-19T03:03:30.573 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:30.573 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:03:30.574 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:30.574 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.660851Z", 2020-11-19T03:03:30.574 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:30.575 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:30.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:30.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:30.575 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:03:30.576 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:30.576 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:30.576 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:30.577 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:30.577 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:03:30.577 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:30.578 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:03:30.578 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:30.579 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:30.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:30.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:30.580 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:03:30.580 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:30.580 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:30.580 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:30.581 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:30.581 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:03:30.581 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:30.581 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:03:30.581 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:30.582 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:30.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:30.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:30.582 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:03:30.583 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:30.583 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:30.583 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:30.584 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:30.584 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:03:30.584 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:30.584 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:03:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:03:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:03:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:03:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:03:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:30.588 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:03:30.590 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:03:30.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:03:30.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:30.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:30.592 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:30.634 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:30.634 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:30.903 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:03:30.904 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:03:30.904 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:03:30.904 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:03:30.905 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:03:30.905 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:03:30.906 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.906 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:03:30.907 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:30.907 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:03:30.907 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:30.908 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:30.908 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755009.660851_356.1", 2020-11-19T03:03:30.909 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:30.909 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:03:30.909 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:30.910 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.660851Z", 2020-11-19T03:03:30.910 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:30.912 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:03:30.912 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:30.912 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:30.913 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:30.913 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:30.913 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:30.914 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:30.914 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:03:30.914 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.914 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:30.915 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:03:30.915 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:30.915 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:30.915 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:30.916 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:30.916 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:30.916 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:30.916 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:30.916 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:30.917 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:03:30.917 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:30.918 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:30.918 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:30.918 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:30.918 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:30.918 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:30.919 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:30.919 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:30.919 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.919 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.919 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:03:30.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:30.920 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:03:30.920 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:03:30.920 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:03:30.920 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:30.921 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:03:30.922 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:03:31.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:31.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:31.137 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:31.185 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:31.186 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:31.405 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:31.401+0000 7f9bc23e90c0 1 bucket sync caught up with source: 2020-11-19T03:03:31.405 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.37.11, , ] 2020-11-19T03:03:31.405 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.37.11, , ] 2020-11-19T03:03:31.405 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:31.401+0000 7f9bc23e90c0 0 bucket checkpoint complete 2020-11-19T03:03:31.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:31.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:31.434 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:31.696+0000 7f4d610f30c0 1 bucket sync caught up with source: 2020-11-19T03:03:31.697 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.37.11, , ] 2020-11-19T03:03:31.697 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.37.11, , ] 2020-11-19T03:03:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:31.696+0000 7f4d610f30c0 0 bucket checkpoint complete 2020-11-19T03:03:31.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:31.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:31.728 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:31.777 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:31.778 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:32.035 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:32.029+0000 7fe8276200c0 1 bucket sync caught up with source: 2020-11-19T03:03:32.035 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.37.11, , ] 2020-11-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.37.11, , ] 2020-11-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:32.029+0000 7fe8276200c0 0 bucket checkpoint complete 2020-11-19T03:03:32.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:32.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:32.221 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:32.223 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:32.448+0000 7f95c09fc0c0 1 bucket sync caught up with source: 2020-11-19T03:03:33.016 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000002.46.5, , ] 2020-11-19T03:03:33.017 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000002.46.5, , ] 2020-11-19T03:03:33.017 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:32.448+0000 7f95c09fc0c0 0 bucket checkpoint complete 2020-11-19T03:03:33.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:33.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:33.019 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:33.110 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:33.111 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:33.342 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:33.340+0000 7f52861050c0 1 bucket sync caught up with source: 2020-11-19T03:03:33.342 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000002.46.5, , ] 2020-11-19T03:03:33.343 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000002.46.5, , ] 2020-11-19T03:03:33.343 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:33.340+0000 7f52861050c0 0 bucket checkpoint complete 2020-11-19T03:03:33.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:33.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:33.375 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:33.464 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:33.464 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:33.701 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:33.700+0000 7f02d78f20c0 1 bucket sync caught up with source: 2020-11-19T03:03:33.702 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000002.60.5, , ] 2020-11-19T03:03:33.702 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000002.60.5, , ] 2020-11-19T03:03:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:33.700+0000 7f02d78f20c0 0 bucket checkpoint complete 2020-11-19T03:03:33.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:33.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:33.729 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:33.778 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:33.778 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:34.022 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:34.017+0000 7f85796100c0 1 bucket sync caught up with source: 2020-11-19T03:03:34.023 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.60.5, , ] 2020-11-19T03:03:34.023 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.60.5, , ] 2020-11-19T03:03:34.023 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:34.017+0000 7f85796100c0 0 bucket checkpoint complete 2020-11-19T03:03:34.054 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-21 zones={test-zone2, test-zone1} 2020-11-19T03:03:34.064 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:34.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.116 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:34.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:34.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:34.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:34.269 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-21 zones={test-zone2, test-zone1} 2020-11-19T03:03:34.270 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-21 zones={test-zone3, test-zone1} 2020-11-19T03:03:34.277 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:34.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.323 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:34.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:34.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:34.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:34.468 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-21 zones={test-zone3, test-zone1} 2020-11-19T03:03:34.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-21 zones={test-zone3, test-zone2} 2020-11-19T03:03:34.479 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:34.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.531 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:34.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:34.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:34.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:34.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:34.675 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-21 zones={test-zone3, test-zone2} 2020-11-19T03:03:34.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:34.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:34.677 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:34.724 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:34.724 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:34.952 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:34.944+0000 7f950f6160c0 1 bucket sync caught up with source: 2020-11-19T03:03:34.952 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2020-11-19T03:03:34.953 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2020-11-19T03:03:34.953 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:34.944+0000 7f950f6160c0 0 bucket checkpoint complete 2020-11-19T03:03:34.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:34.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:34.989 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:35.043 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:35.043 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:35.260 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:35.255+0000 7fdb7c6080c0 1 bucket sync caught up with source: 2020-11-19T03:03:35.260 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2020-11-19T03:03:35.260 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2020-11-19T03:03:35.260 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:35.255+0000 7fdb7c6080c0 0 bucket checkpoint complete 2020-11-19T03:03:35.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:35.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:35.285 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:35.336 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:35.337 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:35.568+0000 7f5c8c40d0c0 1 bucket sync caught up with source: 2020-11-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2020-11-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2020-11-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:35.568+0000 7f5c8c40d0c0 0 bucket checkpoint complete 2020-11-19T03:03:35.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:35.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:36.019+0000 7fd11fb100c0 1 bucket sync caught up with source: 2020-11-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.29.11, , ] 2020-11-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.29.11, , ] 2020-11-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:36.019+0000 7fd11fb100c0 0 bucket checkpoint complete 2020-11-19T03:03:36.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:36.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:36.335+0000 7fac7ab3a0c0 1 bucket sync caught up with source: 2020-11-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.29.11, , ] 2020-11-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.29.11, , ] 2020-11-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:36.335+0000 7fac7ab3a0c0 0 bucket checkpoint complete 2020-11-19T03:03:36.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:36.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:36.707 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:36.703+0000 7f6fd2d630c0 1 bucket sync caught up with source: 2020-11-19T03:03:36.708 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:36.708 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:36.708 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:36.703+0000 7f6fd2d630c0 0 bucket checkpoint complete 2020-11-19T03:03:36.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:36.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:36.735 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:36.783 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:36.783 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:37.026 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:37.020+0000 7fe640df70c0 1 bucket sync caught up with source: 2020-11-19T03:03:37.026 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:37.027 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:37.027 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:37.020+0000 7fe640df70c0 0 bucket checkpoint complete 2020-11-19T03:03:37.058 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-22 zones={test-zone2, test-zone1} 2020-11-19T03:03:37.072 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:37.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.127 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:37.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:37.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:37.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:37.330 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-22 zones={test-zone2, test-zone1} 2020-11-19T03:03:37.439 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-22 zones={test-zone3, test-zone1} 2020-11-19T03:03:37.455 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:37.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.505 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:37.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:37.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:37.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:37.713 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-22 zones={test-zone3, test-zone1} 2020-11-19T03:03:37.713 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-22 zones={test-zone3, test-zone2} 2020-11-19T03:03:37.729 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:37.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.781 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:37.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:37.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:37.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:37.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:37.970 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-22 zones={test-zone3, test-zone2} 2020-11-19T03:03:37.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:37.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:37.971 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:38.027 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:38.028 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:38.274 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:38.268+0000 7f33a03aa0c0 1 bucket sync caught up with source: 2020-11-19T03:03:38.275 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:38.275 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:38.276 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:38.268+0000 7f33a03aa0c0 0 bucket checkpoint complete 2020-11-19T03:03:38.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:38.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:38.302 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:38.351 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:38.351 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:38.599 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:38.595+0000 7f556e99a0c0 1 bucket sync caught up with source: 2020-11-19T03:03:38.599 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:38.599 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:38.600 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:38.595+0000 7f556e99a0c0 0 bucket checkpoint complete 2020-11-19T03:03:38.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:38.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:38.626 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:38.678 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:38.678 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:38.916 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:38.908+0000 7fe55f9200c0 1 bucket sync caught up with source: 2020-11-19T03:03:38.916 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:38.917 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.24.5, , ] 2020-11-19T03:03:38.917 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:38.908+0000 7fe55f9200c0 0 bucket checkpoint complete 2020-11-19T03:03:38.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:38.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:38.945 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:39.231 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:39.227+0000 7fc09b77b0c0 1 bucket sync caught up with source: 2020-11-19T03:03:39.232 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000002.31.5, , ] 2020-11-19T03:03:39.232 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000002.31.5, , ] 2020-11-19T03:03:39.232 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:39.227+0000 7fc09b77b0c0 0 bucket checkpoint complete 2020-11-19T03:03:39.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:39.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:39.264 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:39.310 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:39.310 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:39.541 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:39.539+0000 7fd7aed600c0 1 bucket sync caught up with source: 2020-11-19T03:03:39.542 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000002.31.5, , ] 2020-11-19T03:03:39.542 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000002.31.5, , ] 2020-11-19T03:03:39.542 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:39.539+0000 7fd7aed600c0 0 bucket checkpoint complete 2020-11-19T03:03:39.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:39.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:39.574 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:39.619 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:39.620 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:39.871+0000 7fccc11a50c0 1 bucket sync caught up with source: 2020-11-19T03:03:39.873 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.52.11, , ] 2020-11-19T03:03:39.874 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.52.11, , ] 2020-11-19T03:03:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:39.871+0000 7fccc11a50c0 0 bucket checkpoint complete 2020-11-19T03:03:39.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:39.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:39.901 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:39.950 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:39.951 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:40.174 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:40.168+0000 7f71c8b5e0c0 1 bucket sync caught up with source: 2020-11-19T03:03:40.174 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.52.11, , ] 2020-11-19T03:03:40.175 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.52.11, , ] 2020-11-19T03:03:40.175 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:40.168+0000 7f71c8b5e0c0 0 bucket checkpoint complete 2020-11-19T03:03:40.208 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-23 zones={test-zone2, test-zone1} 2020-11-19T03:03:40.220 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:40.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.267 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:40.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:40.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:40.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:40.527 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-23 zones={test-zone2, test-zone1} 2020-11-19T03:03:40.528 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-23 zones={test-zone3, test-zone1} 2020-11-19T03:03:40.541 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:40.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.593 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:40.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:40.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:40.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:40.760 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-23 zones={test-zone3, test-zone1} 2020-11-19T03:03:40.761 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-23 zones={test-zone3, test-zone2} 2020-11-19T03:03:40.774 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:40.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.823 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:40.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:40.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:40.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:40.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:40.985 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-23 zones={test-zone3, test-zone2} 2020-11-19T03:03:40.986 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-11-19T03:03:40.987 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-24 2020-11-19T03:03:41.022 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-25 2020-11-19T03:03:41.078 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-26 2020-11-19T03:03:41.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:41.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:41.260 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:41.310 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:41.310 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:41.494 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T03:03:41.528 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:03:41.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:41.529 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755021.248167_396.1", 2020-11-19T03:03:41.529 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:03:41.248167Z" 2020-11-19T03:03:41.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:41.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:41.530 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:41.530 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T03:03:41.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:41.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:41.531 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:41.531 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:03:41.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:41.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:41.531 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:41.532 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:03:41.532 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:41.532 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:03:41.533 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:03:41.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:41.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:41.535 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:41.581 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:41.582 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:41.848 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:03:41.848 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:03:41.848 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:03:41.848 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:03:41.849 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:03:41.849 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:03:41.849 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:41.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:41.850 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:03:41.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:41.850 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:03:41.850 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:41.851 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:41.851 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755021.248167_396.1", 2020-11-19T03:03:41.851 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:41.851 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:03:41.851 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:41.852 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.248167Z", 2020-11-19T03:03:41.852 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:41.852 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:41.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:41.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:41.853 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:03:41.853 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:41.853 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:41.853 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:41.854 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:41.854 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:41.854 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:41.854 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:03:41.854 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:41.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:41.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:41.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:41.855 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:03:41.855 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:41.856 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:41.856 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:41.856 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:41.856 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:41.857 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:41.857 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:41.857 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:41.857 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:41.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:41.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:41.858 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:03:41.858 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:41.858 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:41.858 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:41.859 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:41.859 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:41.859 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:41.859 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:41.859 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:41.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:41.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:41.860 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:03:41.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:41.860 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:03:41.861 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:03:41.861 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:03:41.861 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:41.861 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:03:41.862 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:03:41.863 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:03:41.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:41.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:41.864 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:41.906 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:41.907 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:42.160 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:03:42.160 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:03:42.160 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:03:42.161 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:03:42.161 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:03:42.161 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:03:42.161 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:42.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:42.162 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:03:42.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:42.162 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:03:42.162 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:42.163 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:42.163 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755021.248167_396.1", 2020-11-19T03:03:42.163 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:42.163 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:03:42.163 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:42.164 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.248167Z", 2020-11-19T03:03:42.164 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:42.164 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:42.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:42.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:42.165 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:03:42.165 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:42.165 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:42.165 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:42.166 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:42.166 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:03:42.166 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:42.166 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:03:42.167 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:42.167 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:42.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:42.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:42.167 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:03:42.168 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:42.168 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:42.174 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:42.174 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:42.174 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:03:42.175 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:42.175 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:03:42.176 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:42.176 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:42.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:42.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:42.177 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:03:42.178 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:42.178 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:42.178 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:42.179 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:42.179 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:03:42.180 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:42.180 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:03:42.180 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:42.180 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:42.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:42.181 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:03:42.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:42.181 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:03:42.182 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:03:42.182 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:03:42.182 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:42.182 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:03:42.183 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:03:42.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:03:42.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:42.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:42.185 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:42.232 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:42.232 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:42.596 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:03:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:03:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:03:43.670 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:03:43.670 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:03:43.670 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:03:43.670 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:43.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:43.671 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:03:43.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:43.671 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:03:43.671 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755021.248167_396.1", 2020-11-19T03:03:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:03:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.248167Z", 2020-11-19T03:03:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:03:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:03:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:43.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:43.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:43.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:43.676 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:03:43.676 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:43.677 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:43.677 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:43.677 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:43.677 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:43.677 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:43.678 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:43.678 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:43.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:43.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:43.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:43.679 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:03:43.679 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:43.679 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:43.679 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:43.679 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:43.680 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:43.680 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:43.680 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:43.680 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:43.680 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:43.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:43.681 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:03:43.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:43.681 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:03:43.682 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:03:43.682 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:03:43.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:43.682 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:03:43.683 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:03:43.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:43.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:43.951 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:44.002 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:44.002 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:44.224 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:44.216+0000 7f457b53d0c0 1 bucket sync caught up with source: 2020-11-19T03:03:44.224 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.124.6, , ] 2020-11-19T03:03:44.224 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.124.6, , ] 2020-11-19T03:03:44.225 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:44.216+0000 7f457b53d0c0 0 bucket checkpoint complete 2020-11-19T03:03:44.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:44.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:44.251 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:44.305 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:44.305 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:44.548 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:44.543+0000 7f76a60b30c0 1 bucket sync caught up with source: 2020-11-19T03:03:44.549 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.124.6, , ] 2020-11-19T03:03:44.549 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.124.6, , ] 2020-11-19T03:03:44.549 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:44.543+0000 7f76a60b30c0 0 bucket checkpoint complete 2020-11-19T03:03:44.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:44.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:44.576 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:44.624 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:44.625 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:44.878 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:44.868+0000 7feb4e6060c0 1 bucket sync caught up with source: 2020-11-19T03:03:44.878 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.124.6, , ] 2020-11-19T03:03:44.879 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.124.6, , ] 2020-11-19T03:03:44.879 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:44.868+0000 7feb4e6060c0 0 bucket checkpoint complete 2020-11-19T03:03:45.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:45.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:45.003 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:45.054 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:45.054 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:45.298 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:45.295+0000 7f5ed74f40c0 1 bucket sync caught up with source: 2020-11-19T03:03:45.298 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.125.12, , ] 2020-11-19T03:03:45.299 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.125.12, , ] 2020-11-19T03:03:45.299 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:45.295+0000 7f5ed74f40c0 0 bucket checkpoint complete 2020-11-19T03:03:45.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:45.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:45.341 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:45.389 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:45.390 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:45.622 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:45.619+0000 7f11ef63f0c0 1 bucket sync caught up with source: 2020-11-19T03:03:45.623 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.125.12, , ] 2020-11-19T03:03:45.623 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.125.12, , ] 2020-11-19T03:03:45.623 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:45.619+0000 7f11ef63f0c0 0 bucket checkpoint complete 2020-11-19T03:03:45.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:45.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:45.663 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:45.712 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:45.712 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:45.947+0000 7fb1f1f720c0 1 bucket sync caught up with source: 2020-11-19T03:03:45.951 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.131.12, , ] 2020-11-19T03:03:45.951 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.131.12, , ] 2020-11-19T03:03:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:45.947+0000 7fb1f1f720c0 0 bucket checkpoint complete 2020-11-19T03:03:45.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:45.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:45.977 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:46.024 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:46.025 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:46.253 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:46.244+0000 7fdaed2020c0 1 bucket sync caught up with source: 2020-11-19T03:03:46.253 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.131.12, , ] 2020-11-19T03:03:46.253 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.131.12, , ] 2020-11-19T03:03:46.253 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:46.244+0000 7fdaed2020c0 0 bucket checkpoint complete 2020-11-19T03:03:46.279 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-24 zones={test-zone2, test-zone1} 2020-11-19T03:03:46.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:46.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:46.388 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:46.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:46.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:46.488 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-24 zones={test-zone2, test-zone1} 2020-11-19T03:03:46.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-24 zones={test-zone3, test-zone1} 2020-11-19T03:03:46.503 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:46.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:46.556 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:46.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:46.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:46.626 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-24 zones={test-zone3, test-zone1} 2020-11-19T03:03:46.627 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-24 zones={test-zone3, test-zone2} 2020-11-19T03:03:46.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:46.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:46.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:46.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:46.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:46.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-24 zones={test-zone3, test-zone2} 2020-11-19T03:03:46.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:46.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:46.764 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:46.808 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:46.809 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:47.049 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:47.044+0000 7f049ed060c0 1 bucket sync caught up with source: 2020-11-19T03:03:47.050 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.29.12, , ] 2020-11-19T03:03:47.050 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.29.12, , ] 2020-11-19T03:03:47.050 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:47.044+0000 7f049ed060c0 0 bucket checkpoint complete 2020-11-19T03:03:47.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:47.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:47.084 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:47.138 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:47.139 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:47.369 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:47.367+0000 7fdb3d21b0c0 1 bucket sync caught up with source: 2020-11-19T03:03:47.703 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.29.12, , ] 2020-11-19T03:03:47.704 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.29.12, , ] 2020-11-19T03:03:47.704 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:47.367+0000 7fdb3d21b0c0 0 bucket checkpoint complete 2020-11-19T03:03:47.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:47.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:47.706 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:47.757 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:47.757 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:47.993 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:47.988+0000 7f673972d0c0 1 bucket sync caught up with source: 2020-11-19T03:03:47.994 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.29.12, , ] 2020-11-19T03:03:47.994 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.29.12, , ] 2020-11-19T03:03:47.994 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:47.988+0000 7f673972d0c0 0 bucket checkpoint complete 2020-11-19T03:03:48.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:48.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:48.029 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:48.080 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:48.081 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:48.318 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:48.311+0000 7fbf48cc30c0 1 bucket sync caught up with source: 2020-11-19T03:03:48.318 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.39.6, , ] 2020-11-19T03:03:48.318 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.39.6, , ] 2020-11-19T03:03:48.318 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:48.311+0000 7fbf48cc30c0 0 bucket checkpoint complete 2020-11-19T03:03:48.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:48.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:48.349 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:48.681 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:48.679+0000 7f417d26a0c0 1 bucket sync caught up with source: 2020-11-19T03:03:48.682 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.39.6, , ] 2020-11-19T03:03:48.682 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.39.6, , ] 2020-11-19T03:03:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:48.679+0000 7f417d26a0c0 0 bucket checkpoint complete 2020-11-19T03:03:48.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:48.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:48.715 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:48.806 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:48.807 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:49.059 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:49.055+0000 7f92d7fd70c0 1 bucket sync caught up with source: 2020-11-19T03:03:49.059 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.45.12, , ] 2020-11-19T03:03:49.059 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.45.12, , ] 2020-11-19T03:03:49.060 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:49.055+0000 7f92d7fd70c0 0 bucket checkpoint complete 2020-11-19T03:03:49.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:49.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:49.087 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:49.135 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:49.135 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:49.385 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:49.380+0000 7f03350470c0 1 bucket sync caught up with source: 2020-11-19T03:03:49.385 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.45.12, , ] 2020-11-19T03:03:49.385 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.45.12, , ] 2020-11-19T03:03:49.385 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:49.380+0000 7f03350470c0 0 bucket checkpoint complete 2020-11-19T03:03:49.414 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-25 zones={test-zone2, test-zone1} 2020-11-19T03:03:49.426 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:49.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:49.475 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:49.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:49.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:49.552 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-25 zones={test-zone2, test-zone1} 2020-11-19T03:03:49.552 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-25 zones={test-zone3, test-zone1} 2020-11-19T03:03:49.562 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:49.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:49.615 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:49.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:49.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:49.683 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-25 zones={test-zone3, test-zone1} 2020-11-19T03:03:49.684 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-25 zones={test-zone3, test-zone2} 2020-11-19T03:03:49.695 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:49.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:49.747 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:49.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:49.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:49.824 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-25 zones={test-zone3, test-zone2} 2020-11-19T03:03:49.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:49.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:49.825 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:49.874 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:49.875 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:50.119 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:50.112+0000 7fb19c9670c0 1 bucket sync caught up with source: 2020-11-19T03:03:50.119 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2020-11-19T03:03:50.119 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2020-11-19T03:03:50.120 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:50.112+0000 7fb19c9670c0 0 bucket checkpoint complete 2020-11-19T03:03:50.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:50.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:50.149 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:50.210 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:50.210 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:50.444 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:50.439+0000 7f8cb98110c0 1 bucket sync caught up with source: 2020-11-19T03:03:50.444 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2020-11-19T03:03:50.445 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2020-11-19T03:03:50.445 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:50.439+0000 7f8cb98110c0 0 bucket checkpoint complete 2020-11-19T03:03:50.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:50.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:50.470 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:50.524 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:50.526 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:50.773 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:50.768+0000 7f16c715b0c0 1 bucket sync caught up with source: 2020-11-19T03:03:50.773 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2020-11-19T03:03:50.774 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2020-11-19T03:03:50.774 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:50.768+0000 7f16c715b0c0 0 bucket checkpoint complete 2020-11-19T03:03:50.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:50.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:50.898 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:50.947 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:50.947 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:51.173 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:51.167+0000 7fbe9cdd60c0 1 bucket sync caught up with source: 2020-11-19T03:03:51.175 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.15.12, , ] 2020-11-19T03:03:51.176 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.15.12, , ] 2020-11-19T03:03:51.177 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:51.167+0000 7fbe9cdd60c0 0 bucket checkpoint complete 2020-11-19T03:03:51.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:51.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:51.207 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:51.258 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:51.259 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:51.486 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:51.483+0000 7f8681d210c0 1 bucket sync caught up with source: 2020-11-19T03:03:51.486 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000001.15.12, , ] 2020-11-19T03:03:51.487 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000001.15.12, , ] 2020-11-19T03:03:51.487 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:51.483+0000 7f8681d210c0 0 bucket checkpoint complete 2020-11-19T03:03:51.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:51.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:51.515 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:51.563 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:51.563 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:51.805 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:51.803+0000 7fb45c01d0c0 1 bucket sync caught up with source: 2020-11-19T03:03:51.806 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , , , , 00000000003.24.6, , ] 2020-11-19T03:03:51.806 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , , , , 00000000003.24.6, , ] 2020-11-19T03:03:51.806 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:03:51.803+0000 7fb45c01d0c0 0 bucket checkpoint complete 2020-11-19T03:03:51.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:51.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:03:51.838 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:03:51.891 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:51.892 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:52.129 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:52.124+0000 7f8ae97640c0 1 bucket sync caught up with source: 2020-11-19T03:03:52.130 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.24.6, , ] 2020-11-19T03:03:52.130 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.24.6, , ] 2020-11-19T03:03:52.130 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:03:52.124+0000 7f8ae97640c0 0 bucket checkpoint complete 2020-11-19T03:03:52.159 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-26 zones={test-zone2, test-zone1} 2020-11-19T03:03:52.178 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:52.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:52.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:52.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:52.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:53.791 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-26 zones={test-zone2, test-zone1} 2020-11-19T03:03:53.792 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-26 zones={test-zone3, test-zone1} 2020-11-19T03:03:53.803 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:53.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:53.851 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:53.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:53.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:53.923 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-26 zones={test-zone3, test-zone1} 2020-11-19T03:03:53.923 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-26 zones={test-zone3, test-zone2} 2020-11-19T03:03:53.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:03:53.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:53.983 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:03:53.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:03:54.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:03:54.051 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-26 zones={test-zone3, test-zone2} 2020-11-19T03:03:54.052 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-11-19T03:03:54.053 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-27 2020-11-19T03:03:54.086 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-28 2020-11-19T03:03:54.145 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-29 2020-11-19T03:03:54.414 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-11-19T03:03:54.415 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-30 2020-11-19T03:03:54.458 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-31 2020-11-19T03:03:54.511 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-32 2020-11-19T03:03:54.774 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-11-19T03:03:54.775 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-33 2020-11-19T03:03:54.811 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-34 2020-11-19T03:03:54.897 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-35 2020-11-19T03:03:57.163 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-11-19T03:03:57.164 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-36 2020-11-19T03:03:57.202 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-37 2020-11-19T03:03:57.309 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-38 2020-11-19T03:03:58.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:58.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:58.630 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:58.672 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:58.672 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:58.873 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T03:03:58.906 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:03:58.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:58.906 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755038.605369_480.1", 2020-11-19T03:03:58.907 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:03:58.605369Z" 2020-11-19T03:03:58.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:58.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:58.907 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:58.908 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T03:03:58.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:58.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:58.908 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:58.908 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:03:58.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:58.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:58.909 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:58.909 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:03:58.909 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:58.910 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:03:58.910 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:03:58.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:58.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:58.912 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:58.955 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:58.955 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:59.218 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:03:59.219 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:03:59.219 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:03:59.219 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:03:59.220 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:03:59.220 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:03:59.220 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.220 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:03:59.221 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:59.221 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:03:59.221 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:59.221 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:59.222 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755038.605369_480.1", 2020-11-19T03:03:59.222 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:59.222 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:03:59.222 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:59.222 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.605369Z", 2020-11-19T03:03:59.223 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.223 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:59.223 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:03:59.224 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:59.224 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:59.224 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:59.224 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:59.224 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:59.225 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:59.225 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:03:59.225 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.225 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:59.226 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:03:59.226 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:59.227 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:59.227 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:59.227 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:59.227 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:59.227 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:59.228 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:59.228 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.228 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.229 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:59.229 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:03:59.229 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:59.229 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:59.230 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:59.230 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:59.230 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:59.230 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:59.230 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:59.231 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.231 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.231 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.231 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:03:59.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.232 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:03:59.232 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:03:59.232 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:03:59.232 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.233 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:03:59.234 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:03:59.234 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:03:59.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:59.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:59.236 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:59.544 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:03:59.545 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:03:59.545 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:03:59.546 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:03:59.546 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:03:59.547 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:03:59.547 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:59.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:59.548 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:03:59.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:59.549 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:03:59.549 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:59.549 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:59.550 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755038.605369_480.1", 2020-11-19T03:03:59.550 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:59.551 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:03:59.551 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:59.551 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:58.605369Z", 2020-11-19T03:03:59.552 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:59.552 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:59.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:59.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:59.553 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:03:59.554 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:59.554 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:59.555 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:59.555 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:59.555 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:03:59.556 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:59.556 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:03:59.557 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:59.557 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:59.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:59.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:59.558 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:03:59.558 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:59.558 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:59.558 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:59.559 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:59.559 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:03:59.559 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:59.559 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:03:59.559 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:59.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:59.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:59.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:03:59.560 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:03:59.560 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:03:59.561 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:03:59.561 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:03:59.561 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:03:59.561 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:03:59.561 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:03:59.562 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:03:59.562 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:03:59.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:59.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:59.563 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:03:59.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:03:59.563 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:03:59.563 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:03:59.563 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:03:59.564 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:03:59.564 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:03:59.565 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:03:59.565 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:03:59.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:59.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:03:59.567 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:03:59.606 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:03:59.607 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:03:59.873 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:03:59.873 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:03:59.873 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:03:59.873 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:03:59.874 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:03:59.874 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:03:59.874 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.875 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:03:59.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:59.875 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:03:59.876 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:59.876 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:59.876 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755038.605369_480.1", 2020-11-19T03:03:59.877 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:59.877 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:03:59.878 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:59.878 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.605369Z", 2020-11-19T03:03:59.878 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.879 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:59.879 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:03:59.880 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:59.880 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:59.881 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:03:59.882 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:59.882 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:59.882 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:59.882 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:03:59.883 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.883 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:59.883 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:03:59.884 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:59.884 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:59.884 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:59.884 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:59.884 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:59.885 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:59.885 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:59.885 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:03:59.886 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:03:59.886 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:03:59.886 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:03:59.886 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:03:59.887 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:03:59.887 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:03:59.887 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:03:59.887 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:03:59.888 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:03:59.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.888 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:03:59.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:03:59.889 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:03:59.889 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:03:59.889 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:03:59.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:03:59.889 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:03:59.890 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:04:00.290 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-11-19T03:04:00.292 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-39 2020-11-19T03:04:00.341 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-40 2020-11-19T03:04:00.405 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-41 2020-11-19T03:04:00.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:04:00.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:04:00.467 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:04:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:04:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:04:00.708 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=b2c8afed-318c-484c-8dd4-daab7a757391 2020-11-19T03:04:00.743 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:04:00.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:04:00.744 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755040.443579_496.1", 2020-11-19T03:04:00.744 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:04:00.443579Z" 2020-11-19T03:04:00.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:04:00.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:04:00.744 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:04:00.745 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:55:44.294580Z" 2020-11-19T03:04:00.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:04:00.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:04:00.745 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:04:00.746 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:04:00.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:04:00.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:04:00.746 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:04:00.746 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:04:00.747 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:04:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:04:00.748 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:04:00.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:04:00.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:04:00.749 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:04:00.791 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:04:00.791 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:04:01.039 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:04:01.039 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:04:01.039 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:04:01.039 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:04:01.040 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:04:01.040 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:04:01.040 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.041 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:04:01.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:01.041 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:04:01.041 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:04:01.041 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:04:01.042 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755040.443579_496.1", 2020-11-19T03:04:01.042 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:04:01.042 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:04:01.042 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:04:01.042 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.443579Z", 2020-11-19T03:04:01.043 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.043 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:01.043 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:04:01.044 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:04:01.044 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:04:01.044 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:04:01.044 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:04:01.044 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:04:01.045 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:04:01.045 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:04:01.045 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:01.046 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:04:01.046 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:04:01.046 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:04:01.047 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:04:01.047 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:04:01.047 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:04:01.047 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:04:01.047 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:04:01.048 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.048 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:01.048 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:04:01.049 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:04:01.049 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:04:01.049 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:04:01.049 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:04:01.049 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:04:01.050 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:04:01.050 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:04:01.050 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.050 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.050 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.051 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:04:01.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.051 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:04:01.051 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:04:01.051 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:04:01.052 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.052 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:04:01.053 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:04:01.053 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:04:01.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:04:01.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:04:01.055 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:04:01.116 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:04:01.122 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:04:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:04:01.361 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:04:01.361 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:04:01.362 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:04:01.362 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:04:01.362 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:04:01.362 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:04:01.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:04:01.363 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:04:01.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:04:01.363 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:04:01.363 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:04:01.363 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:04:01.364 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755040.443579_496.1", 2020-11-19T03:04:01.364 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:04:01.364 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:04:01.364 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:04:01.364 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.443579Z", 2020-11-19T03:04:01.365 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:04:01.365 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:04:01.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:04:01.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:04:01.366 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:04:01.366 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:04:01.366 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:04:01.366 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:04:01.366 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:04:01.367 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:04:01.367 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:04:01.367 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:04:01.367 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:04:01.367 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:04:01.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:04:01.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:04:01.368 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:04:01.368 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:04:01.369 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:04:01.369 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:04:01.369 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:04:01.369 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:04:01.369 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:04:01.370 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:04:01.370 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:04:01.370 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:04:01.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:04:01.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:04:01.371 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:04:01.371 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:04:01.371 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:04:01.372 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:04:01.372 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:04:01.372 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:04:01.372 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:04:01.372 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:04:01.373 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:04:01.373 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:04:01.373 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:04:01.373 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:04:01.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:04:01.374 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:04:01.374 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:04:01.374 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:04:01.374 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:04:01.375 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:04:01.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:04:01.376 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:04:01.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:04:01.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:04:01.378 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:04:01.418 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:04:01.418 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:04:01.685 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:04:01.686 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:04:01.686 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:04:01.686 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:04:01.686 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:04:01.687 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:04:01.687 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.687 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:04:01.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:01.688 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:04:01.688 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:04:01.688 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:04:01.688 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755040.443579_496.1", 2020-11-19T03:04:01.688 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:04:01.689 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:04:01.689 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:04:01.689 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.443579Z", 2020-11-19T03:04:01.689 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.689 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:01.690 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:04:01.690 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:04:01.691 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:04:01.691 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:04:01.691 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:04:01.691 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:04:01.691 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:04:01.692 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:04:01.692 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.692 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:01.692 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:04:01.693 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:04:01.693 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:04:01.693 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:04:01.693 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:04:01.693 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:04:01.694 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:04:01.694 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:04:01.694 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.694 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:01.695 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:04:01.695 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:04:01.695 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:04:01.695 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:04:01.696 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:04:01.696 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:04:01.696 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:04:01.696 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:04:01.696 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:04:01.697 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.697 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.697 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:04:01.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:01.697 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:04:01.698 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:04:01.698 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:04:01.698 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:01.698 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:04:01.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:04:01.700 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-19T03:04:01.702 INFO:teuthology.orchestra.run:waiting for 300 2020-11-19T03:04:01.702 INFO:tasks.rgw.c1.client.1.smithi086.stdout:2020-11-19T03:04:01.699+0000 7f8e0f7fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 15956) UID: 0 2020-11-19T03:04:01.703 INFO:tasks.rgw.c1.client.1.smithi086.stdout:2020-11-19T03:04:01.699+0000 7f8e2b778100 -1 shutting down 2020-11-19T03:04:07.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-19T03:04:07.804 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-19T03:04:07.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:04:07.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:04:07.806 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:04:07.845 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:04:07.846 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:04:07.871 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:04:07.864+0000 7fe38a7210c0 0 NOTICE: overriding master zone: d71482d6-15fd-4715-9992-a25958dcb538 2020-11-19T03:04:07.876 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:04:07.868+0000 7fe38a7210c0 0 failed reading obj info from .rgw.root:zone_info.d71482d6-15fd-4715-9992-a25958dcb538: (2) No such file or directory 2020-11-19T03:04:07.876 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:04:07.868+0000 7fe38a7210c0 0 WARNING: could not read zone params for zone id=d71482d6-15fd-4715-9992-a25958dcb538 name=test-zone1 2020-11-19T03:04:07.877 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:04:07.872+0000 7fe38a7210c0 0 failed reading obj info from .rgw.root:zone_info.fdf730f5-4390-45ac-9d84-6887e17cb661: (2) No such file or directory 2020-11-19T03:04:07.877 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:04:07.872+0000 7fe38a7210c0 0 WARNING: could not read zone params for zone id=fdf730f5-4390-45ac-9d84-6887e17cb661 name=test-zone3 2020-11-19T03:04:07.888 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:04:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:04:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:04:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-19T03:04:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-19T03:04:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-19T03:04:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-19T03:04:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-19T03:04:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-19T03:04:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-19T03:04:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-19T03:04:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-19T03:04:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-19T03:04:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-19T03:04:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-19T03:04:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-19T03:04:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-19T03:04:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T03:04:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T03:04:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T03:04:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T03:04:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T03:04:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:04:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-19T03:04:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T03:04:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T03:04:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-19T03:04:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2020-11-19T03:04:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-19T03:04:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T03:04:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:04:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-19T03:04:07.896 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:04:07.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:04:07.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:04:07.898 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:04:07.981 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:04:07.981 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:04:08.272 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:04:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:04:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 1, 2020-11-19T03:04:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "b2c8afed-318c-484c-8dd4-daab7a757391", 2020-11-19T03:04:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2020-11-19T03:04:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1605755040.443579_496.1", 2020-11-19T03:04:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1605754544.294580_119.1", 2020-11-19T03:04:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:04:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2020-11-19T03:04:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2020-11-19T03:04:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:04:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2020-11-19T03:04:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2020-11-19T03:04:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2020-11-19T03:04:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2020-11-19T03:04:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": "true", 2020-11-19T03:04:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:04:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:04:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2020-11-19T03:04:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2020-11-19T03:04:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:04:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2020-11-19T03:04:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:04:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2020-11-19T03:04:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:04:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:04:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:04:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:04:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:04:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:04:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2020-11-19T03:04:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "false", 2020-11-19T03:04:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2020-11-19T03:04:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2020-11-19T03:04:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:04:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:04:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:04:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:04:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:04:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:04:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:04:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:04:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:04:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:04:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:04:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:04:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:04:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:04:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2020-11-19T03:04:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:04:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:04:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:04:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:04:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:04:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:04:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:04:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:04:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:04:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:04:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:04:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2020-11-19T03:04:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:04:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:04:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:04:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:04:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:04:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:04:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:04:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:04:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:04:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:04:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2020-11-19T03:04:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2020-11-19T03:04:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2020-11-19T03:04:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2020-11-19T03:04:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2020-11-19T03:04:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:04:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2020-11-19T03:04:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:04:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2020-11-19T03:04:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2020-11-19T03:04:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:04:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2020-11-19T03:04:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:04:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 1446762810 2020-11-19T03:04:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:04:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2049584274 2020-11-19T03:04:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:04:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2043733610 2020-11-19T03:04:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:04:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:04:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2362875045 2020-11-19T03:04:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:04:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:04:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:04:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2020-11-19T03:04:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2020-11-19T03:04:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:04:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:04:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:04:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:04:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:04:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2020-11-19T03:04:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:04:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:04:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:04:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:04:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:04:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:04:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:04:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:04:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2020-11-19T03:04:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 3 2020-11-19T03:04:08.322 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:04:08.324 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-11-19T03:04:12.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-19T03:05:08.326 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-42 2020-11-19T03:05:08.460 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-43 2020-11-19T03:05:08.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:05:08.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:05:08.501 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:05:08.539 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:05:08.540 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:05:08.759 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=0e78ea22-1fd1-48d9-9be5-2a207091b0ca 2020-11-19T03:05:08.792 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:05:08.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:05:08.793 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755108.494399_209.1", 2020-11-19T03:05:08.793 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:05:08.494399Z" 2020-11-19T03:05:08.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:05:08.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:05:08.794 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:05:08.794 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2020-11-19T03:05:08.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:05:08.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:05:08.795 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:05:08.795 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2020-11-19T03:05:08.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:05:08.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:05:08.796 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:05:08.796 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2020-11-19T03:05:08.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:05:08.796 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:05:08.797 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-11-19T03:05:08.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:05:08.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:05:08.799 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:05:08.843 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:05:08.843 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:05:09.109 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:05:09.109 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:05:09.109 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:05:09.110 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:05:09.110 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:05:09.110 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:05:09.110 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 3 2020-11-19T03:05:09.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.111 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:05:09.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.111 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:05:09.111 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:05:09.111 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:05:09.112 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755108.494399_209.1", 2020-11-19T03:05:09.112 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:05:09.112 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 80, 2020-11-19T03:05:09.112 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:05:09.113 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.494399Z", 2020-11-19T03:05:09.113 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 3 2020-11-19T03:05:09.113 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.114 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:05:09.114 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:05:09.114 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:05:09.114 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:05:09.115 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:05:09.115 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:05:09.115 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:05:09.115 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:05:09.115 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 3 2020-11-19T03:05:09.116 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.116 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:05:09.116 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:05:09.117 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:05:09.117 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:05:09.117 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:05:09.117 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:05:09.117 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:05:09.118 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:05:09.118 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 3 2020-11-19T03:05:09.118 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.119 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:05:09.119 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:05:09.119 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:05:09.120 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:05:09.120 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:05:09.120 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:05:09.120 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:05:09.120 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:05:09.121 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 3 2020-11-19T03:05:09.121 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.121 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.121 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:05:09.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.122 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:05:09.122 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 80, 2020-11-19T03:05:09.122 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 80 2020-11-19T03:05:09.122 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.122 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:05:09.124 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-11-19T03:05:09.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:05:09.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:05:09.126 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:05:09.208 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:05:09.209 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:05:09.236 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:05:09.230+0000 7f2b819ee0c0 0 NOTICE: overriding master zone: c9a90d7d-dd43-4d9d-9b12-b4d640ca300c 2020-11-19T03:05:09.237 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:05:09.234+0000 7f2b819ee0c0 0 failed reading obj info from .rgw.root:zone_info.0c377913-e2b8-4ccf-b637-8d38a8727f98: (2) No such file or directory 2020-11-19T03:05:09.237 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:05:09.234+0000 7f2b819ee0c0 0 WARNING: could not read zone params for zone id=0c377913-e2b8-4ccf-b637-8d38a8727f98 name=test-zone4 2020-11-19T03:05:09.237 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:05:09.234+0000 7f2b819ee0c0 0 failed reading obj info from .rgw.root:zone_info.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c: (2) No such file or directory 2020-11-19T03:05:09.238 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:05:09.234+0000 7f2b819ee0c0 0 WARNING: could not read zone params for zone id=c9a90d7d-dd43-4d9d-9b12-b4d640ca300c name=test-zone2 2020-11-19T03:05:09.248 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:05:09.248 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:05:09.249 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T03:05:09.249 INFO:teuthology.orchestra.run.smithi086.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-19T03:05:09.249 INFO:teuthology.orchestra.run.smithi086.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-19T03:05:09.249 INFO:teuthology.orchestra.run.smithi086.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-19T03:05:09.250 INFO:teuthology.orchestra.run.smithi086.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-19T03:05:09.250 INFO:teuthology.orchestra.run.smithi086.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-19T03:05:09.250 INFO:teuthology.orchestra.run.smithi086.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-19T03:05:09.250 INFO:teuthology.orchestra.run.smithi086.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-19T03:05:09.250 INFO:teuthology.orchestra.run.smithi086.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-19T03:05:09.251 INFO:teuthology.orchestra.run.smithi086.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-19T03:05:09.251 INFO:teuthology.orchestra.run.smithi086.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-19T03:05:09.251 INFO:teuthology.orchestra.run.smithi086.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-19T03:05:09.251 INFO:teuthology.orchestra.run.smithi086.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-19T03:05:09.252 INFO:teuthology.orchestra.run.smithi086.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-19T03:05:09.252 INFO:teuthology.orchestra.run.smithi086.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-19T03:05:09.252 INFO:teuthology.orchestra.run.smithi086.stdout: "system_key": { 2020-11-19T03:05:09.252 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T03:05:09.252 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T03:05:09.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.253 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_pools": [ 2020-11-19T03:05:09.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.253 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "default-placement", 2020-11-19T03:05:09.253 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:05:09.254 INFO:teuthology.orchestra.run.smithi086.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-19T03:05:09.254 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": { 2020-11-19T03:05:09.254 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD": { 2020-11-19T03:05:09.254 INFO:teuthology.orchestra.run.smithi086.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-19T03:05:09.255 INFO:teuthology.orchestra.run.smithi086.stdout: "compression_type": "random" 2020-11-19T03:05:09.255 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.255 INFO:teuthology.orchestra.run.smithi086.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-19T03:05:09.255 INFO:teuthology.orchestra.run.smithi086.stdout: "index_type": 0 2020-11-19T03:05:09.256 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.256 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.256 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.256 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:05:09.256 INFO:teuthology.orchestra.run.smithi086.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-19T03:05:09.257 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:05:09.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:05:09.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:05:09.259 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:05:09.342 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:05:09.342 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:05:09.647 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:05:09.647 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:05:09.647 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2020-11-19T03:05:09.648 INFO:teuthology.orchestra.run.smithi086.stdout: "predecessor_uuid": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:05:09.648 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": [ 2020-11-19T03:05:09.648 INFO:teuthology.orchestra.run.smithi086.stdout: "1_1605755108.494399_209.1", 2020-11-19T03:05:09.648 INFO:teuthology.orchestra.run.smithi086.stdout: "", 2020-11-19T03:05:09.648 INFO:teuthology.orchestra.run.smithi086.stdout: "", 2020-11-19T03:05:09.649 INFO:teuthology.orchestra.run.smithi086.stdout: "" 2020-11-19T03:05:09.649 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.649 INFO:teuthology.orchestra.run.smithi086.stdout: "period_map": { 2020-11-19T03:05:09.649 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:05:09.649 INFO:teuthology.orchestra.run.smithi086.stdout: "zonegroups": [ 2020-11-19T03:05:09.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.650 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "test-zonegroup", 2020-11-19T03:05:09.650 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zonegroup", 2020-11-19T03:05:09.650 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "", 2020-11-19T03:05:09.651 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "true", 2020-11-19T03:05:09.651 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:05:09.651 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:05:09.651 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.651 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T03:05:09.652 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T03:05:09.652 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:05:09.652 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [ 2020-11-19T03:05:09.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.652 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:05:09.653 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone4", 2020-11-19T03:05:09.653 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:05:09.653 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:05:09.653 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.653 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:05:09.654 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:05:09.654 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:05:09.654 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:05:09.654 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "pubsub", 2020-11-19T03:05:09.654 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "false", 2020-11-19T03:05:09.655 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [ 2020-11-19T03:05:09.655 INFO:teuthology.orchestra.run.smithi086.stdout: "test-zone1" 2020-11-19T03:05:09.655 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.655 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:05:09.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.656 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:05:09.656 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone2", 2020-11-19T03:05:09.656 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:05:09.656 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:05:09.657 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.657 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:05:09.657 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:05:09.657 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:05:09.657 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:05:09.658 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:05:09.658 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:05:09.658 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:05:09.658 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:05:09.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.659 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:05:09.659 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T03:05:09.659 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:05:09.660 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:05:09.660 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.660 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:05:09.660 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:05:09.660 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:05:09.661 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:05:09.661 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:05:09.661 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:05:09.661 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:05:09.661 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:05:09.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.662 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:05:09.662 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone3", 2020-11-19T03:05:09.662 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:05:09.663 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:05:09.663 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.664 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:05:09.664 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:05:09.664 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:05:09.664 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:05:09.665 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:05:09.665 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:05:09.665 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:05:09.665 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:05:09.665 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.666 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.666 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [ 2020-11-19T03:05:09.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.666 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default-placement", 2020-11-19T03:05:09.666 INFO:teuthology.orchestra.run.smithi086.stdout: "tags": [], 2020-11-19T03:05:09.667 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": [ 2020-11-19T03:05:09.667 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD" 2020-11-19T03:05:09.667 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:05:09.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.667 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.668 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "default-placement", 2020-11-19T03:05:09.668 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:05:09.668 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T03:05:09.668 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T03:05:09.668 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.669 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:05:09.675 INFO:teuthology.orchestra.run.smithi086.stdout: "short_zone_ids": [ 2020-11-19T03:05:09.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.675 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:05:09.675 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 1446762810 2020-11-19T03:05:09.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.676 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:05:09.676 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2049584274 2020-11-19T03:05:09.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.677 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:05:09.677 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2043733610 2020-11-19T03:05:09.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:05:09.678 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:05:09.678 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2362875045 2020-11-19T03:05:09.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.678 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:05:09.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.679 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:05:09.679 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:05:09.679 INFO:teuthology.orchestra.run.smithi086.stdout: "period_config": { 2020-11-19T03:05:09.679 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2020-11-19T03:05:09.680 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T03:05:09.680 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T03:05:09.680 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T03:05:09.680 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T03:05:09.680 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T03:05:09.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.681 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2020-11-19T03:05:09.681 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T03:05:09.681 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T03:05:09.681 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T03:05:09.682 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T03:05:09.682 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T03:05:09.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:05:09.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:05:09.682 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:05:09.683 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_name": "test-realm", 2020-11-19T03:05:09.683 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:05:09.683 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:05:09.685 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-11-19T03:06:09.686 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-44 2020-11-19T03:06:09.806 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-45 2020-11-19T03:06:09.890 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-19T03:06:09.891 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-19T03:06:09.893 INFO:tasks.rgw.c1.client.1:Started 2020-11-19T03:06:09.894 INFO:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:8001 2020-11-19T03:06:09.911 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:06:09.911 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:06:09.914 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi086.front.sepia.ceph.com port 8001: Connection refused 2020-11-19T03:06:09.915 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T03:06:10.917 INFO:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:8001 2020-11-19T03:06:10.938 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:06:10.939 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:06:10.945 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-19T03:06:10.946 INFO:teuthology.orchestra.run.smithi086.stdout:anonymous 2020-11-19T03:06:10.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:10.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:10.948 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:11.049 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:11.050 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:11.273 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:06:11.304 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755169.859139_400.1", 2020-11-19T03:06:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:06:09.859139Z" 2020-11-19T03:06:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:06:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:06:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:06:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:11.310 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:11.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:06:11.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:11.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:11.313 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:11.350 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:11.351 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:11.589 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:06:11.590 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:06:11.590 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:06:11.590 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:06:11.590 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:06:11.591 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:06:11.591 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:11.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:11.592 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:06:11.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:11.593 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:06:11.593 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:11.593 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:11.594 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755169.859139_400.1", 2020-11-19T03:06:11.594 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:11.594 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:06:11.595 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:11.595 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.859139Z", 2020-11-19T03:06:11.595 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:11.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:11.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:11.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:11.596 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:06:11.597 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:11.597 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:11.597 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:06:11.597 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:11.598 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:11.598 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:11.598 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:06:11.599 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:11.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:11.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:11.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:11.600 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:06:11.600 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:11.600 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:11.601 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:11.601 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:11.602 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:11.602 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:11.602 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:11.603 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:11.603 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:11.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:11.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:11.604 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:06:11.604 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:11.604 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:11.604 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:11.605 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:11.605 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:11.605 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:11.605 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:11.605 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:11.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:11.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:11.606 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:06:11.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:11.606 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:06:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:06:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:06:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:11.607 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:06:11.608 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:06:11.609 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:06:11.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:11.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:11.610 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:11.652 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:11.652 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:11.912 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:06:11.913 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:06:11.914 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:06:11.915 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:06:11.915 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:06:11.916 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:06:11.916 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:06:11.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:11.919 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:06:11.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:11.920 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:06:11.920 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:11.921 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:11.921 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755169.859139_400.1", 2020-11-19T03:06:11.922 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:11.922 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:06:11.922 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:11.923 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.859139Z", 2020-11-19T03:06:11.923 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:06:11.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:11.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:11.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:11.925 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:06:11.925 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:11.925 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:11.926 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:06:11.927 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:11.927 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:11.929 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:11.929 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:06:11.930 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:11.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:11.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:11.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:11.931 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:06:11.933 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:11.933 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:11.934 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:11.935 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:11.935 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:11.936 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:11.936 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:06:11.937 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:11.937 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:11.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:11.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:11.944 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:06:11.944 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:11.944 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:11.945 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:11.945 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:11.946 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:11.946 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:11.947 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:06:11.947 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:11.948 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:11.949 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:11.950 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:06:11.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:11.952 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:06:11.953 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:06:11.953 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:06:11.954 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:11.957 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:06:11.959 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:06:11.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:06:11.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:11.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:11.967 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:12.023 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:12.024 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:12.290 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:06:12.291 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:06:12.291 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:06:12.291 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:06:12.291 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:06:12.292 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:06:12.292 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:12.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:12.293 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:06:12.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:12.293 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:06:12.294 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:12.294 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:12.295 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755169.859139_400.1", 2020-11-19T03:06:12.295 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:12.296 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:06:12.296 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:12.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.859139Z", 2020-11-19T03:06:12.297 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:12.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:12.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:12.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:12.298 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:06:12.298 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:12.298 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:12.299 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:06:12.299 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:12.299 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:12.300 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:12.300 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:06:12.300 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:12.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:12.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:12.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:12.302 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:06:12.302 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:12.302 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:12.303 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:12.303 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:12.303 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:12.303 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:12.304 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:12.304 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:12.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:12.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:12.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:12.305 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:06:12.306 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:12.306 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:12.306 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:12.307 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:12.307 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:12.307 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:12.307 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:12.308 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:12.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:12.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:12.309 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:06:12.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:12.309 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:06:12.309 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:06:12.310 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:06:12.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:12.310 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:06:12.312 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:06:12.313 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-39 zones={test-zone2, test-zone1} 2020-11-19T03:06:12.336 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:12.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:12.472 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-39 zones={test-zone2, test-zone1} 2020-11-19T03:06:12.474 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-39 zones={test-zone3, test-zone1} 2020-11-19T03:06:12.517 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:12.582 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:12.675 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-39 zones={test-zone3, test-zone1} 2020-11-19T03:06:12.676 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-39 zones={test-zone1, test-zone2} 2020-11-19T03:06:12.688 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:12.738 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:12.812 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-39 zones={test-zone1, test-zone2} 2020-11-19T03:06:13.540 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-39 zones={test-zone3, test-zone2} 2020-11-19T03:06:13.556 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:13.612 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:13.688 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-39 zones={test-zone3, test-zone2} 2020-11-19T03:06:13.688 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-39 zones={test-zone1, test-zone3} 2020-11-19T03:06:13.702 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:13.752 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:13.825 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-39 zones={test-zone1, test-zone3} 2020-11-19T03:06:13.825 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-39 zones={test-zone2, test-zone3} 2020-11-19T03:06:13.839 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:13.890 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:13.963 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-39 zones={test-zone2, test-zone3} 2020-11-19T03:06:13.963 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-40 zones={test-zone2, test-zone1} 2020-11-19T03:06:13.981 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:14.034 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:14.104 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-40 zones={test-zone2, test-zone1} 2020-11-19T03:06:14.105 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-40 zones={test-zone3, test-zone1} 2020-11-19T03:06:14.115 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:14.166 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:14.233 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-40 zones={test-zone3, test-zone1} 2020-11-19T03:06:14.233 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-40 zones={test-zone1, test-zone2} 2020-11-19T03:06:14.244 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:14.290 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:14.356 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-40 zones={test-zone1, test-zone2} 2020-11-19T03:06:14.356 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-40 zones={test-zone3, test-zone2} 2020-11-19T03:06:14.365 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:14.414 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:14.477 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-40 zones={test-zone3, test-zone2} 2020-11-19T03:06:14.477 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-40 zones={test-zone1, test-zone3} 2020-11-19T03:06:14.488 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:14.534 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:14.592 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-40 zones={test-zone1, test-zone3} 2020-11-19T03:06:14.592 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-40 zones={test-zone2, test-zone3} 2020-11-19T03:06:14.602 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:14.650 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:14.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-40 zones={test-zone2, test-zone3} 2020-11-19T03:06:14.719 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-41 zones={test-zone2, test-zone1} 2020-11-19T03:06:14.735 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:14.786 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:14.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-41 zones={test-zone2, test-zone1} 2020-11-19T03:06:14.850 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-41 zones={test-zone3, test-zone1} 2020-11-19T03:06:14.861 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:14.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:14.972 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-41 zones={test-zone3, test-zone1} 2020-11-19T03:06:14.973 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-41 zones={test-zone1, test-zone2} 2020-11-19T03:06:14.985 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:15.034 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:15.102 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-41 zones={test-zone1, test-zone2} 2020-11-19T03:06:15.103 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-41 zones={test-zone3, test-zone2} 2020-11-19T03:06:15.113 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:15.162 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:15.225 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-41 zones={test-zone3, test-zone2} 2020-11-19T03:06:15.225 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-41 zones={test-zone1, test-zone3} 2020-11-19T03:06:15.239 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:15.286 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:15.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-41 zones={test-zone1, test-zone3} 2020-11-19T03:06:15.344 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-41 zones={test-zone2, test-zone3} 2020-11-19T03:06:15.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:15.402 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:15.459 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-41 zones={test-zone2, test-zone3} 2020-11-19T03:06:15.460 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-42 zones={test-zone2, test-zone1} 2020-11-19T03:06:15.469 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:15.518 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:15.580 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-42 zones={test-zone2, test-zone1} 2020-11-19T03:06:15.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-42 zones={test-zone3, test-zone1} 2020-11-19T03:06:15.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:15.634 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:15.707 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-42 zones={test-zone3, test-zone1} 2020-11-19T03:06:15.708 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-42 zones={test-zone1, test-zone2} 2020-11-19T03:06:15.721 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:15.773 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:15.858 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-42 zones={test-zone1, test-zone2} 2020-11-19T03:06:15.859 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-42 zones={test-zone3, test-zone2} 2020-11-19T03:06:15.871 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:15.919 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:15.987 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-42 zones={test-zone3, test-zone2} 2020-11-19T03:06:15.988 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-42 zones={test-zone1, test-zone3} 2020-11-19T03:06:16.000 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:16.047 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:16.115 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-42 zones={test-zone1, test-zone3} 2020-11-19T03:06:16.115 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-42 zones={test-zone2, test-zone3} 2020-11-19T03:06:16.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:16.174 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:16.246 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-42 zones={test-zone2, test-zone3} 2020-11-19T03:06:16.247 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-43 zones={test-zone2, test-zone1} 2020-11-19T03:06:16.258 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:16.310 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:16.373 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-43 zones={test-zone2, test-zone1} 2020-11-19T03:06:16.374 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-43 zones={test-zone3, test-zone1} 2020-11-19T03:06:16.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:16.434 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:16.497 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-43 zones={test-zone3, test-zone1} 2020-11-19T03:06:16.498 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-43 zones={test-zone1, test-zone2} 2020-11-19T03:06:16.509 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:16.554 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:16.617 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-43 zones={test-zone1, test-zone2} 2020-11-19T03:06:16.618 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-43 zones={test-zone3, test-zone2} 2020-11-19T03:06:16.629 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:16.678 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:16.741 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-43 zones={test-zone3, test-zone2} 2020-11-19T03:06:16.742 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-43 zones={test-zone1, test-zone3} 2020-11-19T03:06:16.754 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:16.807 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:16.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-43 zones={test-zone1, test-zone3} 2020-11-19T03:06:16.895 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-43 zones={test-zone2, test-zone3} 2020-11-19T03:06:16.913 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:16.970 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:17.048 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-43 zones={test-zone2, test-zone3} 2020-11-19T03:06:17.049 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-44 zones={test-zone2, test-zone1} 2020-11-19T03:06:17.064 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:17.121 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:17.194 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-44 zones={test-zone2, test-zone1} 2020-11-19T03:06:17.194 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-44 zones={test-zone3, test-zone1} 2020-11-19T03:06:17.205 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:17.255 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:17.325 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-44 zones={test-zone3, test-zone1} 2020-11-19T03:06:17.326 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-44 zones={test-zone1, test-zone2} 2020-11-19T03:06:17.338 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:17.386 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:17.472 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-44 zones={test-zone1, test-zone2} 2020-11-19T03:06:17.473 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-44 zones={test-zone3, test-zone2} 2020-11-19T03:06:17.492 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:17.553 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:17.659 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-44 zones={test-zone3, test-zone2} 2020-11-19T03:06:18.820 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-44 zones={test-zone1, test-zone3} 2020-11-19T03:06:18.840 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:18.890 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:18.955 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-44 zones={test-zone1, test-zone3} 2020-11-19T03:06:18.955 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-44 zones={test-zone2, test-zone3} 2020-11-19T03:06:18.969 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:19.019 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:19.093 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-44 zones={test-zone2, test-zone3} 2020-11-19T03:06:19.094 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-45 zones={test-zone2, test-zone1} 2020-11-19T03:06:19.108 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:19.158 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:19.226 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-45 zones={test-zone2, test-zone1} 2020-11-19T03:06:19.227 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-45 zones={test-zone3, test-zone1} 2020-11-19T03:06:19.236 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:19.286 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:19.348 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-45 zones={test-zone3, test-zone1} 2020-11-19T03:06:19.349 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-45 zones={test-zone1, test-zone2} 2020-11-19T03:06:19.360 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:19.414 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:19.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-45 zones={test-zone1, test-zone2} 2020-11-19T03:06:19.479 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-45 zones={test-zone3, test-zone2} 2020-11-19T03:06:19.490 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:19.538 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:19.605 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-45 zones={test-zone3, test-zone2} 2020-11-19T03:06:19.606 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-45 zones={test-zone1, test-zone3} 2020-11-19T03:06:19.617 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:19.666 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:19.734 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-45 zones={test-zone1, test-zone3} 2020-11-19T03:06:19.734 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-45 zones={test-zone2, test-zone3} 2020-11-19T03:06:19.744 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:06:19.794 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:06:19.858 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-45 zones={test-zone2, test-zone3} 2020-11-19T03:06:19.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:19.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:19.859 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b2c8afed-318c-484c-8dd4-daab7a757391 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:19.895 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:19.896 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:20.131 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:20.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.132 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754414.632290_106.1", 2020-11-19T03:06:20.132 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "user", 2020-11-19T03:06:20.132 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "rgw-multisite-test-user", 2020-11-19T03:06:20.132 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.632290363Z", 2020-11-19T03:06:20.132 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.133 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.133 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.133 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.134 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.134 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_OD787OEqCb37r-dWrIu2W6g", 2020-11-19T03:06:20.134 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.134 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.135 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.135 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.135 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.136 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754414.640760_109.1", 2020-11-19T03:06:20.136 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "user", 2020-11-19T03:06:20.136 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "rgw-multisite-test-user", 2020-11-19T03:06:20.136 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.640760721Z", 2020-11-19T03:06:20.136 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.137 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.137 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.137 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.137 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.138 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_OD787OEqCb37r-dWrIu2W6g", 2020-11-19T03:06:20.138 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.138 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.138 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.139 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.755439_114.1", 2020-11-19T03:06:20.140 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.140 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10", 2020-11-19T03:06:20.140 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.755439437Z", 2020-11-19T03:06:20.140 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.140 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jDQMSP1imZk9M53g_LemGsD", 2020-11-19T03:06:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.759756_127.1", 2020-11-19T03:06:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10", 2020-11-19T03:06:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.759756660Z", 2020-11-19T03:06:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jDQMSP1imZk9M53g_LemGsD", 2020-11-19T03:06:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.761697_128.1", 2020-11-19T03:06:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-10", 2020-11-19T03:06:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.761697955Z", 2020-11-19T03:06:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_QaR_8xY7tB3j-5_fwfgIAp2", 2020-11-19T03:06:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.765274_129.1", 2020-11-19T03:06:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-10", 2020-11-19T03:06:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.765274847Z", 2020-11-19T03:06:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_QaR_8xY7tB3j-5_fwfgIAp2", 2020-11-19T03:06:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.789431_130.1", 2020-11-19T03:06:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11", 2020-11-19T03:06:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.789431860Z", 2020-11-19T03:06:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_h4zkJ0TD5OOIK8n5LNF_3YA", 2020-11-19T03:06:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.158 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.158 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.158 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.159 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.793774_131.1", 2020-11-19T03:06:20.159 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.159 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11", 2020-11-19T03:06:20.159 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.793774959Z", 2020-11-19T03:06:20.159 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.160 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.160 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.160 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.160 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.161 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_h4zkJ0TD5OOIK8n5LNF_3YA", 2020-11-19T03:06:20.161 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.161 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.161 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.162 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.162 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.162 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.798353_132.1", 2020-11-19T03:06:20.163 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.163 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-11", 2020-11-19T03:06:20.163 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.798353113Z", 2020-11-19T03:06:20.163 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.163 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.164 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.164 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.164 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.164 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5lbvhy-2ssHalMo0XAyb9jK", 2020-11-19T03:06:20.165 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.165 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.165 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.165 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.166 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.166 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.803735_133.1", 2020-11-19T03:06:20.166 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.167 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-11", 2020-11-19T03:06:20.167 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.803735217Z", 2020-11-19T03:06:20.167 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.167 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.167 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.168 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.168 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.168 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5lbvhy-2ssHalMo0XAyb9jK", 2020-11-19T03:06:20.168 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.169 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.169 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.169 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.169 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.170 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.854454_134.1", 2020-11-19T03:06:20.170 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.170 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12", 2020-11-19T03:06:20.171 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.854454685Z", 2020-11-19T03:06:20.171 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.171 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.171 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.172 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.172 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.172 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_DG8ZKomIxnXa3vMXcvP8ajO", 2020-11-19T03:06:20.172 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.173 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.173 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.173 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.173 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.174 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.858973_135.1", 2020-11-19T03:06:20.174 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.174 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12", 2020-11-19T03:06:20.175 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.858973052Z", 2020-11-19T03:06:20.175 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.175 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.175 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.175 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.176 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.176 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_DG8ZKomIxnXa3vMXcvP8ajO", 2020-11-19T03:06:20.176 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.177 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.177 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.177 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.177 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.178 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.860848_136.1", 2020-11-19T03:06:20.178 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.178 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-12", 2020-11-19T03:06:20.178 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.860848274Z", 2020-11-19T03:06:20.179 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.179 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.179 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.179 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.180 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.180 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_xHh-uu7OMNHE3E1tkdnCoN0", 2020-11-19T03:06:20.180 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.180 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.181 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.182 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.866167_137.1", 2020-11-19T03:06:20.182 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.182 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-12", 2020-11-19T03:06:20.182 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.866167085Z", 2020-11-19T03:06:20.182 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.183 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.183 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.183 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.183 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.184 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_xHh-uu7OMNHE3E1tkdnCoN0", 2020-11-19T03:06:20.184 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.184 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.184 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.787642_145.1", 2020-11-19T03:06:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13", 2020-11-19T03:06:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.787642625Z", 2020-11-19T03:06:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qxTzHkbF9wemeZVynRNqeCj", 2020-11-19T03:06:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.792198_150.1", 2020-11-19T03:06:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13", 2020-11-19T03:06:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.792198045Z", 2020-11-19T03:06:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qxTzHkbF9wemeZVynRNqeCj", 2020-11-19T03:06:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.794166_151.1", 2020-11-19T03:06:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-13", 2020-11-19T03:06:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.794166672Z", 2020-11-19T03:06:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ogTrNysaHhYexwepXbm3dwI", 2020-11-19T03:06:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.798068_152.1", 2020-11-19T03:06:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-13", 2020-11-19T03:06:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.798068026Z", 2020-11-19T03:06:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ogTrNysaHhYexwepXbm3dwI", 2020-11-19T03:06:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.819538_154.1", 2020-11-19T03:06:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14", 2020-11-19T03:06:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.819538205Z", 2020-11-19T03:06:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dcIW3-alzbWV6sgq13MRCG7", 2020-11-19T03:06:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.825038_155.1", 2020-11-19T03:06:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14", 2020-11-19T03:06:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.825038088Z", 2020-11-19T03:06:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dcIW3-alzbWV6sgq13MRCG7", 2020-11-19T03:06:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.827341_156.1", 2020-11-19T03:06:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-14", 2020-11-19T03:06:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.827341897Z", 2020-11-19T03:06:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_TbVBTXwBuNJEI3zmckRRIjz", 2020-11-19T03:06:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.832699_157.1", 2020-11-19T03:06:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-14", 2020-11-19T03:06:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.832699383Z", 2020-11-19T03:06:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_TbVBTXwBuNJEI3zmckRRIjz", 2020-11-19T03:06:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.868186_158.1", 2020-11-19T03:06:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15", 2020-11-19T03:06:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.868186115Z", 2020-11-19T03:06:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Pzb_ZFMDCfzlSj1lsKyhj_z", 2020-11-19T03:06:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.872546_159.1", 2020-11-19T03:06:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15", 2020-11-19T03:06:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.872546446Z", 2020-11-19T03:06:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Pzb_ZFMDCfzlSj1lsKyhj_z", 2020-11-19T03:06:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.876242_160.1", 2020-11-19T03:06:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-15", 2020-11-19T03:06:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.876242156Z", 2020-11-19T03:06:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Hkg8dq-5IxwuJ-1F2ioM3Ha", 2020-11-19T03:06:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.888825_161.1", 2020-11-19T03:06:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-15", 2020-11-19T03:06:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.888825992Z", 2020-11-19T03:06:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Hkg8dq-5IxwuJ-1F2ioM3Ha", 2020-11-19T03:06:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.231 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.231 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.231 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.231 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.798543_207.1", 2020-11-19T03:06:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.798543679Z", 2020-11-19T03:06:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.803204_211.1", 2020-11-19T03:06:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.803204912Z", 2020-11-19T03:06:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.805294_212.1", 2020-11-19T03:06:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16", 2020-11-19T03:06:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.805294677Z", 2020-11-19T03:06:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_8SdF_vsimBqHGDaqiMgkarR", 2020-11-19T03:06:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.810747_213.1", 2020-11-19T03:06:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16", 2020-11-19T03:06:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.810747502Z", 2020-11-19T03:06:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_8SdF_vsimBqHGDaqiMgkarR", 2020-11-19T03:06:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.838151_214.1", 2020-11-19T03:06:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.838151294Z", 2020-11-19T03:06:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.842707_215.1", 2020-11-19T03:06:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.842707380Z", 2020-11-19T03:06:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.844853_216.1", 2020-11-19T03:06:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17", 2020-11-19T03:06:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.844853013Z", 2020-11-19T03:06:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__nKT7kdBFED-s1dk1lgSXVH", 2020-11-19T03:06:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.849738_217.1", 2020-11-19T03:06:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17", 2020-11-19T03:06:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.849738526Z", 2020-11-19T03:06:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__nKT7kdBFED-s1dk1lgSXVH", 2020-11-19T03:06:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.905621_218.1", 2020-11-19T03:06:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.905621496Z", 2020-11-19T03:06:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.911693_219.1", 2020-11-19T03:06:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.911693551Z", 2020-11-19T03:06:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.915252_220.1", 2020-11-19T03:06:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18", 2020-11-19T03:06:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.915252896Z", 2020-11-19T03:06:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6l8yn5mDyDSfq7DcU_4ezuR", 2020-11-19T03:06:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.275 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.919671_221.1", 2020-11-19T03:06:20.275 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18", 2020-11-19T03:06:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.919671168Z", 2020-11-19T03:06:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6l8yn5mDyDSfq7DcU_4ezuR", 2020-11-19T03:06:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.939873_222.1", 2020-11-19T03:06:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.939873846Z", 2020-11-19T03:06:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.281 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.281 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:20.281 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.282 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.282 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.282 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.282 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.283 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.948027_223.1", 2020-11-19T03:06:20.283 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.283 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:20.283 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.948027995Z", 2020-11-19T03:06:20.284 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.284 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.284 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:20.284 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.285 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.285 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:20.285 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.285 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.286 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.287 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.955102_224.1", 2020-11-19T03:06:20.287 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.287 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:20.287 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.955102556Z", 2020-11-19T03:06:20.287 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.288 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.288 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:20.288 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.288 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.289 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:20.289 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.289 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.289 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.290 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.290 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.290 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.962377_225.1", 2020-11-19T03:06:20.291 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.291 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:20.291 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.962377954Z", 2020-11-19T03:06:20.291 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.291 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.292 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:20.292 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.292 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.292 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:20.293 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.293 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.293 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.293 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.294 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.294 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.974309_226.1", 2020-11-19T03:06:20.294 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.295 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:20.295 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.974309142Z", 2020-11-19T03:06:20.295 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.295 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.295 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:20.296 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.296 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.296 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:20.296 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.297 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.297 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.297 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.297 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.983290_227.1", 2020-11-19T03:06:20.298 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.983290859Z", 2020-11-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.299 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:20.300 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.300 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.300 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:20.300 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.853615_253.1", 2020-11-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.302 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:20.303 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.853615363Z", 2020-11-19T03:06:20.303 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.303 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.303 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.303 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.304 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.304 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:20.304 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.304 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.857577_262.1", 2020-11-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:20.306 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.857577608Z", 2020-11-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.308 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.308 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:20.308 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.308 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.309 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.310 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.859745_263.1", 2020-11-19T03:06:20.310 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.310 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19", 2020-11-19T03:06:20.310 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.859745204Z", 2020-11-19T03:06:20.310 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.311 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.311 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.311 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.311 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.312 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Qy4UpA_MiVj8fLm1yoUPwmx", 2020-11-19T03:06:20.312 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.312 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.312 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.313 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.863407_264.1", 2020-11-19T03:06:20.314 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.314 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19", 2020-11-19T03:06:20.314 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.863407887Z", 2020-11-19T03:06:20.314 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.314 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.315 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.315 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.315 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.315 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Qy4UpA_MiVj8fLm1yoUPwmx", 2020-11-19T03:06:20.316 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.316 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.316 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.316 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.317 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.317 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.869018_265.1", 2020-11-19T03:06:20.317 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.318 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:20.318 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.869018103Z", 2020-11-19T03:06:20.318 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.318 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.318 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:20.319 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.319 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.319 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:20.319 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.320 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.320 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.320 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.320 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.321 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.874924_266.1", 2020-11-19T03:06:20.321 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.321 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:20.322 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.874924462Z", 2020-11-19T03:06:20.322 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.322 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.322 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:20.322 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.323 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.323 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:20.323 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.324 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.324 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.325 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754893.333355_282.1", 2020-11-19T03:06:20.325 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.325 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:20.325 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:33.333355116Z", 2020-11-19T03:06:20.326 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.326 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.326 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.326 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.327 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.327 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.327 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.327 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.328 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.329 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754893.338774_293.1", 2020-11-19T03:06:20.329 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.329 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:20.329 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:33.338774111Z", 2020-11-19T03:06:20.329 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.330 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.330 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.330 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.330 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.331 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.331 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.331 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.331 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.332 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754893.341581_294.1", 2020-11-19T03:06:20.333 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.333 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20", 2020-11-19T03:06:20.333 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:33.341581490Z", 2020-11-19T03:06:20.333 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.333 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.334 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.334 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.334 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.334 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_m4q0aNW1r6ONbHC6SDkHRNx", 2020-11-19T03:06:20.335 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.335 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.335 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.336 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.336 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.337 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754893.346605_295.1", 2020-11-19T03:06:20.337 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.337 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20", 2020-11-19T03:06:20.337 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:33.346605721Z", 2020-11-19T03:06:20.338 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.338 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.338 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.338 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.339 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.339 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_m4q0aNW1r6ONbHC6SDkHRNx", 2020-11-19T03:06:20.339 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.339 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.340 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.341 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754898.951630_299.1", 2020-11-19T03:06:20.341 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.341 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:20.341 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:38.951630338Z", 2020-11-19T03:06:20.341 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.342 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.342 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.342 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.342 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.343 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.343 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.343 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.343 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.344 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.344 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.344 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754898.962533_304.1", 2020-11-19T03:06:20.345 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.345 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:20.345 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:38.962533901Z", 2020-11-19T03:06:20.345 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.345 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.346 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.346 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.346 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.346 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.347 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.347 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.347 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.347 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.348 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.348 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754969.539612_314.1", 2020-11-19T03:06:20.348 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.349 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:20.349 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:02:49.539612219Z", 2020-11-19T03:06:20.349 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.349 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.349 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.350 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.350 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.350 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.350 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.352 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754969.549049_315.1", 2020-11-19T03:06:20.352 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.352 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:20.353 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:02:49.549049417Z", 2020-11-19T03:06:20.353 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.353 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.353 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.354 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.354 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.354 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:20.354 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:20.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.355 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.355 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.356 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.481687_339.1", 2020-11-19T03:06:20.356 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.356 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:20.357 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.481687459Z", 2020-11-19T03:06:20.357 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.357 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.357 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.357 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.358 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.358 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:20.358 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.359 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.359 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.359 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.359 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.360 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.486007_340.1", 2020-11-19T03:06:20.360 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.360 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:20.360 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.486007579Z", 2020-11-19T03:06:20.361 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.361 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.361 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.361 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.362 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.362 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:20.362 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.362 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.363 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.363 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.363 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.364 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.487864_341.1", 2020-11-19T03:06:20.364 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.364 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21", 2020-11-19T03:06:20.364 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.487864468Z", 2020-11-19T03:06:20.364 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.365 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.365 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.365 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.365 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.366 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UwTo_benAnYmaa4CegfHe2h", 2020-11-19T03:06:20.366 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.366 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.366 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.367 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.367 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.367 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.492471_342.1", 2020-11-19T03:06:20.368 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.368 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21", 2020-11-19T03:06:20.368 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.492471230Z", 2020-11-19T03:06:20.368 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.368 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.369 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.369 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.369 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.369 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UwTo_benAnYmaa4CegfHe2h", 2020-11-19T03:06:20.370 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.370 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.370 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.371 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.516426_343.1", 2020-11-19T03:06:20.372 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.372 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:20.372 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.516426936Z", 2020-11-19T03:06:20.372 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.372 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.373 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.373 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.373 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.373 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:20.374 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.374 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.374 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.374 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.375 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.520864_344.1", 2020-11-19T03:06:20.375 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.376 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:20.376 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.520864137Z", 2020-11-19T03:06:20.376 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.376 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.376 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.377 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.377 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.377 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:20.378 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.378 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.378 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.379 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.379 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.380 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.522824_345.1", 2020-11-19T03:06:20.380 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.380 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22", 2020-11-19T03:06:20.381 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.522824373Z", 2020-11-19T03:06:20.381 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.381 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.381 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.382 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.382 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.383 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-PCUgWQeHT2U7eFpqPZrnJx", 2020-11-19T03:06:20.383 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.383 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.384 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.527162_346.1", 2020-11-19T03:06:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22", 2020-11-19T03:06:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.527162988Z", 2020-11-19T03:06:20.386 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.386 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.386 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.386 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.387 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.387 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-PCUgWQeHT2U7eFpqPZrnJx", 2020-11-19T03:06:20.387 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.387 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.389 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.572422_347.1", 2020-11-19T03:06:20.389 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.389 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:20.389 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.572422879Z", 2020-11-19T03:06:20.389 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.390 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.390 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.390 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.390 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.391 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:20.391 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.391 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.391 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.392 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.578468_348.1", 2020-11-19T03:06:20.393 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.393 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:20.393 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.578468758Z", 2020-11-19T03:06:20.393 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.393 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.394 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.394 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.394 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.394 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:20.395 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.395 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.395 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.395 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.396 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.396 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.580555_349.1", 2020-11-19T03:06:20.396 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.397 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23", 2020-11-19T03:06:20.397 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.580555073Z", 2020-11-19T03:06:20.397 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.397 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.397 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.398 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.398 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.398 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_PtmtnLQARXBOkzZFBwIKFhM", 2020-11-19T03:06:20.398 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.399 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.399 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.399 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.399 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.400 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.585888_350.1", 2020-11-19T03:06:20.400 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.400 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23", 2020-11-19T03:06:20.401 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.585888316Z", 2020-11-19T03:06:20.401 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.401 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.401 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.401 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.402 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.402 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_PtmtnLQARXBOkzZFBwIKFhM", 2020-11-19T03:06:20.402 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.403 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.403 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.403 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.403 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.404 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.610407_351.1", 2020-11-19T03:06:20.404 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.404 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:20.404 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.610407033Z", 2020-11-19T03:06:20.405 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.405 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.405 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:20.405 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.406 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.406 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:20.406 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.406 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.407 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.407 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.407 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.408 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.616786_352.1", 2020-11-19T03:06:20.408 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.408 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:20.408 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.616786647Z", 2020-11-19T03:06:20.408 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.409 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.409 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:20.409 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.410 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.410 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:20.410 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.410 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.411 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.411 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.411 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.412 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.625234_353.1", 2020-11-19T03:06:20.412 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.412 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:20.412 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.625234108Z", 2020-11-19T03:06:20.412 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.413 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.413 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:20.413 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.413 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.414 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:20.414 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.414 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.414 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.415 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.633175_354.1", 2020-11-19T03:06:20.416 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.416 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:20.416 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.633175833Z", 2020-11-19T03:06:20.416 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.416 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.417 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:20.417 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.417 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.417 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:20.418 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.418 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.418 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.418 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.419 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.419 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.644998_355.1", 2020-11-19T03:06:20.419 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.420 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:20.420 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.644998413Z", 2020-11-19T03:06:20.420 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.420 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.420 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:20.421 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.421 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.421 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:20.421 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.422 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.422 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.422 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.422 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.423 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.660851_356.1", 2020-11-19T03:06:20.423 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.423 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:20.424 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.660851700Z", 2020-11-19T03:06:20.424 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.424 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.424 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:20.424 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.425 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.425 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:20.425 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.426 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.426 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.426 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.426 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.427 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.005483_361.1", 2020-11-19T03:06:20.427 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.427 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:20.427 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.005483628Z", 2020-11-19T03:06:20.428 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.428 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.428 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.428 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.429 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.429 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.429 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.429 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.430 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.430 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.430 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.431 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.009877_374.1", 2020-11-19T03:06:20.431 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.431 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:20.431 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.009877496Z", 2020-11-19T03:06:20.431 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.432 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.432 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.432 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.432 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.433 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.433 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.433 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.433 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.434 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.434 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.434 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.012023_375.1", 2020-11-19T03:06:20.435 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.435 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24", 2020-11-19T03:06:20.435 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.012023407Z", 2020-11-19T03:06:20.435 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.435 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.436 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.436 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.436 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.436 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ycH_LWaCb7pN9H3I9w2Z6N_", 2020-11-19T03:06:20.437 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.437 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.437 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.437 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.438 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.438 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.016947_376.1", 2020-11-19T03:06:20.439 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.439 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24", 2020-11-19T03:06:20.439 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.016947542Z", 2020-11-19T03:06:20.439 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.439 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.440 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.440 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.440 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.440 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ycH_LWaCb7pN9H3I9w2Z6N_", 2020-11-19T03:06:20.441 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.441 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.441 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.441 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.442 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.442 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.045357_377.1", 2020-11-19T03:06:20.442 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.443 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:20.443 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.045357346Z", 2020-11-19T03:06:20.443 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.443 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.443 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.443 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.444 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.444 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.444 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.445 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.445 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.446 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.050883_378.1", 2020-11-19T03:06:20.446 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.446 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:20.446 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.050883011Z", 2020-11-19T03:06:20.447 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.447 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.447 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.447 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.448 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.448 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.448 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.448 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.449 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.450 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.053418_379.1", 2020-11-19T03:06:20.450 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.450 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25", 2020-11-19T03:06:20.450 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.053418708Z", 2020-11-19T03:06:20.451 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.451 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.451 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.451 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.452 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.452 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bkpajY1jnfqrwJDp6NVto29", 2020-11-19T03:06:20.452 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.452 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.453 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.454 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.059396_380.1", 2020-11-19T03:06:20.454 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.454 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25", 2020-11-19T03:06:20.454 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.059396024Z", 2020-11-19T03:06:20.454 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.455 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.455 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.455 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.456 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.456 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bkpajY1jnfqrwJDp6NVto29", 2020-11-19T03:06:20.456 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.456 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.457 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.457 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.457 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.458 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.092006_381.1", 2020-11-19T03:06:20.458 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.458 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:20.458 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.092006149Z", 2020-11-19T03:06:20.458 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.459 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.459 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.459 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.459 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.460 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.460 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.460 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.460 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.461 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.461 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.461 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.096908_382.1", 2020-11-19T03:06:20.462 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.462 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:20.462 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.096908426Z", 2020-11-19T03:06:20.462 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.462 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.463 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.463 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.463 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.463 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.464 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.464 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.464 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.464 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.465 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.465 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.099132_383.1", 2020-11-19T03:06:20.465 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.466 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26", 2020-11-19T03:06:20.466 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.099132448Z", 2020-11-19T03:06:20.466 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.466 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.466 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.467 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.467 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.467 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_WAoNjTvGKPea_y2zejmRi9k", 2020-11-19T03:06:20.467 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.468 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.468 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.468 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.468 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.469 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.109731_384.1", 2020-11-19T03:06:20.469 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.469 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26", 2020-11-19T03:06:20.470 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.109731627Z", 2020-11-19T03:06:20.470 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.470 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.470 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.470 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.471 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.471 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_WAoNjTvGKPea_y2zejmRi9k", 2020-11-19T03:06:20.471 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.472 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.472 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.473 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.128366_385.1", 2020-11-19T03:06:20.473 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.473 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:20.473 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.128366050Z", 2020-11-19T03:06:20.474 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.474 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.474 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.474 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.475 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.475 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.475 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.475 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.476 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.477 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.134273_386.1", 2020-11-19T03:06:20.477 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.477 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:20.477 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.134273445Z", 2020-11-19T03:06:20.477 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.478 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.478 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.478 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.478 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.479 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.479 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.479 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.479 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.480 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.138369_387.1", 2020-11-19T03:06:20.481 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.481 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:20.481 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.138369812Z", 2020-11-19T03:06:20.481 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.481 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.482 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.482 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.482 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.482 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.483 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:20.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.483 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.483 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.483 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.484 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.484 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.151920_388.1", 2020-11-19T03:06:20.484 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.484 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:20.485 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.151920363Z", 2020-11-19T03:06:20.485 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.485 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.485 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.485 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.486 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.486 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:20.486 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:20.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.487 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.487 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.487 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.487 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.488 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.157598_389.1", 2020-11-19T03:06:20.488 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.488 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:20.488 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.157598827Z", 2020-11-19T03:06:20.489 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.489 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.489 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.489 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.490 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.490 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.490 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.490 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.491 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.491 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.491 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.492 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.167497_390.1", 2020-11-19T03:06:20.492 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.492 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:20.492 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.167497429Z", 2020-11-19T03:06:20.493 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.493 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.493 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.493 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.494 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.494 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.494 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.494 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.495 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.495 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.495 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.496 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.178487_391.1", 2020-11-19T03:06:20.496 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.496 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:20.496 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.178487831Z", 2020-11-19T03:06:20.496 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.497 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.497 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.497 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.497 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.498 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.498 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:20.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.498 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.498 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.499 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.499 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.499 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.186130_392.1", 2020-11-19T03:06:20.500 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.500 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:20.500 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.186130781Z", 2020-11-19T03:06:20.500 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.501 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.501 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.501 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.501 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.502 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:20.502 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:20.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.502 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.502 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.503 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.503 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.503 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.200136_393.1", 2020-11-19T03:06:20.504 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.504 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:20.504 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.200136493Z", 2020-11-19T03:06:20.504 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.504 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.505 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.505 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.505 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.505 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.506 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.506 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.506 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.507 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.507 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.507 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.228723_394.1", 2020-11-19T03:06:20.508 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.508 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:20.508 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.228723559Z", 2020-11-19T03:06:20.508 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.508 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.509 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.509 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.509 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.509 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.510 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.510 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.510 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.510 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.511 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.511 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.241089_395.1", 2020-11-19T03:06:20.512 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.512 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:20.512 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.241089363Z", 2020-11-19T03:06:20.512 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.512 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.513 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.513 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.513 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.513 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.514 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:20.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.514 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.514 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.514 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.515 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.515 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.248167_396.1", 2020-11-19T03:06:20.515 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.516 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:20.516 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.248167458Z", 2020-11-19T03:06:20.516 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.516 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.516 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.517 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.517 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.517 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:20.518 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:20.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.518 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.518 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.518 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.519 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.070651_403.1", 2020-11-19T03:06:20.519 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.520 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:20.520 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.070651958Z", 2020-11-19T03:06:20.520 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.520 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.520 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.520 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.521 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.521 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:20.521 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.522 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.522 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.522 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.522 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.523 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.075617_416.1", 2020-11-19T03:06:20.523 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.523 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:20.524 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.075617906Z", 2020-11-19T03:06:20.524 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.524 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.524 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.524 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.525 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.525 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:20.525 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.526 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.526 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.527 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.077978_417.1", 2020-11-19T03:06:20.527 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.527 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27", 2020-11-19T03:06:20.527 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.077978681Z", 2020-11-19T03:06:20.528 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.528 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.528 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.528 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.528 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.529 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jwhmoKKel5WXgN7QZadkAxM", 2020-11-19T03:06:20.529 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.529 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.530 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.530 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.530 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.531 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.081633_418.1", 2020-11-19T03:06:20.531 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.531 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27", 2020-11-19T03:06:20.531 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.081633830Z", 2020-11-19T03:06:20.531 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.532 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.532 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.532 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.532 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.533 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jwhmoKKel5WXgN7QZadkAxM", 2020-11-19T03:06:20.533 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.533 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.533 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.534 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.534 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.534 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.106122_419.1", 2020-11-19T03:06:20.535 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.535 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:20.535 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.106122712Z", 2020-11-19T03:06:20.535 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.535 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.536 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.536 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.536 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.536 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.538 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.111140_420.1", 2020-11-19T03:06:20.538 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.539 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:20.539 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.111140679Z", 2020-11-19T03:06:20.539 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.539 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.539 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.540 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.540 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.540 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:20.540 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.541 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.541 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.542 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.113526_421.1", 2020-11-19T03:06:20.542 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.543 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28", 2020-11-19T03:06:20.543 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.113526370Z", 2020-11-19T03:06:20.543 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.543 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.543 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.543 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.544 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.544 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_BALtCKJ9w_4jnJuGrJpKnG-", 2020-11-19T03:06:20.544 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.545 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.545 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.545 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.545 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.546 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.118961_422.1", 2020-11-19T03:06:20.546 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.546 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28", 2020-11-19T03:06:20.546 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.118961566Z", 2020-11-19T03:06:20.547 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.547 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.547 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.547 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.548 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.548 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_BALtCKJ9w_4jnJuGrJpKnG-", 2020-11-19T03:06:20.548 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.548 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.549 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.549 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.549 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.550 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.167348_423.1", 2020-11-19T03:06:20.550 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.550 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:20.550 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.167348445Z", 2020-11-19T03:06:20.550 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.551 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.551 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.551 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.551 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.552 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:20.552 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.552 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.552 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.553 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.553 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.553 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.172591_424.1", 2020-11-19T03:06:20.554 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.554 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:20.554 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.172591027Z", 2020-11-19T03:06:20.554 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.554 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.555 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.555 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.555 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.555 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:20.556 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.556 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.556 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.557 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.557 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.174666_425.1", 2020-11-19T03:06:20.557 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.558 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29", 2020-11-19T03:06:20.558 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.174666875Z", 2020-11-19T03:06:20.558 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.558 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.558 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.559 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.559 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.559 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7NmprpCcXUiFYkFKdzDwvBW", 2020-11-19T03:06:20.559 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.560 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.560 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.561 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.178503_426.1", 2020-11-19T03:06:20.561 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.561 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29", 2020-11-19T03:06:20.562 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.178503216Z", 2020-11-19T03:06:20.562 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.562 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.562 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.562 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.563 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.563 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7NmprpCcXUiFYkFKdzDwvBW", 2020-11-19T03:06:20.563 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.564 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.564 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.564 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.564 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.565 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.200545_427.1", 2020-11-19T03:06:20.565 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.565 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:20.565 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.200545045Z", 2020-11-19T03:06:20.566 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.566 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.566 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:20.566 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.567 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.567 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:20.567 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.567 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.568 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.571 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.208326_428.1", 2020-11-19T03:06:20.571 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.571 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:20.571 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.208326306Z", 2020-11-19T03:06:20.572 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.572 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.572 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:20.572 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.574 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.574 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.574 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.574 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.257261_429.1", 2020-11-19T03:06:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.257261094Z", 2020-11-19T03:06:20.576 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.576 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.576 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:20.576 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.577 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.577 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:20.577 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.577 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.271719_430.1", 2020-11-19T03:06:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.271719934Z", 2020-11-19T03:06:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.580 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.580 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:20.580 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.580 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.348435_431.1", 2020-11-19T03:06:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.348435126Z", 2020-11-19T03:06:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.584 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:20.584 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.584 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.584 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.358783_432.1", 2020-11-19T03:06:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.358783093Z", 2020-11-19T03:06:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:20.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.588 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.588 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:20.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.590 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.436658_433.1", 2020-11-19T03:06:20.590 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.590 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.436658111Z", 2020-11-19T03:06:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.594 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.443801_434.1", 2020-11-19T03:06:20.594 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.594 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:20.594 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.443801075Z", 2020-11-19T03:06:20.595 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.595 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.595 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.595 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.596 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.596 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:20.596 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.596 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.597 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.597 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.597 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.598 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.447377_435.1", 2020-11-19T03:06:20.598 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.598 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30", 2020-11-19T03:06:20.598 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.447377487Z", 2020-11-19T03:06:20.599 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.599 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.599 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.599 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.600 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.600 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_mM8-TW-E_epjEOhv1WllsAD", 2020-11-19T03:06:20.600 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.600 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.600 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.601 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.601 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.602 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.452011_436.1", 2020-11-19T03:06:20.602 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.602 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30", 2020-11-19T03:06:20.602 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.452011568Z", 2020-11-19T03:06:20.602 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.602 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.603 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.603 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.603 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.603 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_mM8-TW-E_epjEOhv1WllsAD", 2020-11-19T03:06:20.604 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.604 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.604 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.604 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.605 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.605 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.486369_437.1", 2020-11-19T03:06:20.605 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.606 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:20.606 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.486369972Z", 2020-11-19T03:06:20.606 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.606 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.606 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.607 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.607 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.607 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:20.607 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.608 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.608 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.608 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.608 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.609 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.490281_438.1", 2020-11-19T03:06:20.609 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.609 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:20.610 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.490281477Z", 2020-11-19T03:06:20.610 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.610 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.610 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.610 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.611 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.611 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:20.611 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.612 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.612 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.612 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.612 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.613 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.492125_439.1", 2020-11-19T03:06:20.613 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.613 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31", 2020-11-19T03:06:20.614 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.492125730Z", 2020-11-19T03:06:20.614 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.614 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.614 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.614 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.615 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.615 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_gXleTzFvDf4A5fTuHAWH5iB", 2020-11-19T03:06:20.615 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.616 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.616 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.616 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.616 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.495843_440.1", 2020-11-19T03:06:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31", 2020-11-19T03:06:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.495843939Z", 2020-11-19T03:06:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_gXleTzFvDf4A5fTuHAWH5iB", 2020-11-19T03:06:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.528063_441.1", 2020-11-19T03:06:20.621 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.621 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:20.621 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.528063552Z", 2020-11-19T03:06:20.621 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.624 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.532035_442.1", 2020-11-19T03:06:20.624 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.625 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:20.625 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.532035043Z", 2020-11-19T03:06:20.625 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.625 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.626 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.626 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.626 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.626 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:20.627 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.627 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.627 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.627 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.628 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.534111_443.1", 2020-11-19T03:06:20.628 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.628 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32", 2020-11-19T03:06:20.629 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.534111357Z", 2020-11-19T03:06:20.629 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.629 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.629 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.630 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.630 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.630 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_1MkPrP26FSf3qf6aGsjMwfM", 2020-11-19T03:06:20.630 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.631 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.631 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.632 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.544040_444.1", 2020-11-19T03:06:20.632 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.632 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32", 2020-11-19T03:06:20.633 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.544040853Z", 2020-11-19T03:06:20.633 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.633 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.633 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.633 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.634 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.634 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_1MkPrP26FSf3qf6aGsjMwfM", 2020-11-19T03:06:20.634 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.635 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.635 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.635 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.635 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.636 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.575866_445.1", 2020-11-19T03:06:20.636 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.636 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:20.636 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.575866003Z", 2020-11-19T03:06:20.637 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.637 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.637 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:20.637 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.638 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.638 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:20.638 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.638 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.639 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.639 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.639 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.640 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.585216_446.1", 2020-11-19T03:06:20.640 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.640 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:20.640 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.585216241Z", 2020-11-19T03:06:20.640 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.641 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.641 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:20.641 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.641 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.642 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:20.642 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.642 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.642 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.643 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.643 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.643 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.639139_447.1", 2020-11-19T03:06:20.644 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.644 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:20.644 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.639139709Z", 2020-11-19T03:06:20.644 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.644 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.645 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:20.645 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.645 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.645 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:20.646 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.646 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.646 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.646 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.647 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.647 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.649165_448.1", 2020-11-19T03:06:20.647 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.648 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:20.648 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.649165203Z", 2020-11-19T03:06:20.648 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.648 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.648 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:20.649 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.649 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.649 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:20.650 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.650 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.650 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.651 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.651 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.710088_449.1", 2020-11-19T03:06:20.651 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.710088485Z", 2020-11-19T03:06:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.652 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:20.653 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.654 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.720549_450.1", 2020-11-19T03:06:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.655 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:20.656 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.720549878Z", 2020-11-19T03:06:20.656 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.656 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.656 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:20.656 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.657 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.657 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:20.657 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.658 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.658 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.658 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.658 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.659 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.794865_451.1", 2020-11-19T03:06:20.659 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.659 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:20.659 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.794865388Z", 2020-11-19T03:06:20.660 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.660 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.660 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.660 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.661 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.661 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:20.661 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.661 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.662 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.663 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.799032_452.1", 2020-11-19T03:06:20.663 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.663 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:20.663 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.799032992Z", 2020-11-19T03:06:20.663 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.664 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.664 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.664 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.664 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.665 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:20.665 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.665 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.665 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.666 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.801245_453.1", 2020-11-19T03:06:20.667 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.667 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33", 2020-11-19T03:06:20.667 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.801245355Z", 2020-11-19T03:06:20.667 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.667 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.668 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.668 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.668 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.668 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_8lKVoHs2FBMY1x8WlQGeuNM", 2020-11-19T03:06:20.669 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.669 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.669 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.670 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.805695_454.1", 2020-11-19T03:06:20.671 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.671 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33", 2020-11-19T03:06:20.671 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.805695695Z", 2020-11-19T03:06:20.671 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.671 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.672 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.672 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.672 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.672 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_8lKVoHs2FBMY1x8WlQGeuNM", 2020-11-19T03:06:20.673 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.673 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.673 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.674 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.674 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.832169_455.1", 2020-11-19T03:06:20.674 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.675 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:20.675 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.832169972Z", 2020-11-19T03:06:20.675 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.675 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.675 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.676 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.676 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.676 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:20.676 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.677 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.677 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.678 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.837510_456.1", 2020-11-19T03:06:20.678 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.678 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:20.679 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.837510588Z", 2020-11-19T03:06:20.679 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.679 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.679 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.679 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.680 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.680 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:20.680 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.681 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.681 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.681 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.681 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.682 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.839242_457.1", 2020-11-19T03:06:20.682 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.682 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34", 2020-11-19T03:06:20.682 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.839242953Z", 2020-11-19T03:06:20.683 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.683 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.683 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.683 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.684 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.684 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uX_C_amZ6wHaMpUOaDbLhIh", 2020-11-19T03:06:20.684 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.684 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.685 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.685 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.685 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.686 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.843268_458.1", 2020-11-19T03:06:20.686 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.686 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34", 2020-11-19T03:06:20.686 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.843268836Z", 2020-11-19T03:06:20.686 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.687 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.687 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.687 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.687 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.688 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uX_C_amZ6wHaMpUOaDbLhIh", 2020-11-19T03:06:20.688 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.688 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.688 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.689 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.689 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.690 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.918282_459.1", 2020-11-19T03:06:20.690 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.690 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:20.690 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.918282306Z", 2020-11-19T03:06:20.690 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.691 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.691 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.691 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.691 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.692 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:20.692 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.692 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.692 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.693 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.693 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.693 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.927511_460.1", 2020-11-19T03:06:20.694 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.694 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:20.694 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.927511062Z", 2020-11-19T03:06:20.694 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.694 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:20.696 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.696 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.696 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.930242_461.1", 2020-11-19T03:06:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.698 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35", 2020-11-19T03:06:20.698 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.930242283Z", 2020-11-19T03:06:20.698 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.698 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.698 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.699 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.699 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.699 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_LIlIyIKf54pQyVwNBj8qRry", 2020-11-19T03:06:20.699 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.700 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.700 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.701 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.935757_462.1", 2020-11-19T03:06:20.701 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.701 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35", 2020-11-19T03:06:20.702 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.935757929Z", 2020-11-19T03:06:20.702 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.702 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.702 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.702 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.703 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.703 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_LIlIyIKf54pQyVwNBj8qRry", 2020-11-19T03:06:20.703 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.704 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.704 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.704 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.704 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.705 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.960222_463.1", 2020-11-19T03:06:20.705 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.705 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:20.705 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.960222969Z", 2020-11-19T03:06:20.706 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.706 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.706 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:20.706 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.707 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.707 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:20.707 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.707 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.708 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.709 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.966451_464.1", 2020-11-19T03:06:20.709 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.709 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:20.709 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.966451383Z", 2020-11-19T03:06:20.709 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.710 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.710 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:20.710 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.710 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.711 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:20.711 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.711 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.711 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.712 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.712 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.712 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755036.023079_465.1", 2020-11-19T03:06:20.713 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.713 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:20.713 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:56.023079326Z", 2020-11-19T03:06:20.713 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.713 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.714 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:20.714 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.714 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.715 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:20.715 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.715 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.715 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.716 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.716 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.716 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755036.036438_466.1", 2020-11-19T03:06:20.717 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.717 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:20.717 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:56.036438127Z", 2020-11-19T03:06:20.717 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.718 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.718 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:20.718 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.718 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.719 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:20.719 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.719 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.719 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.720 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.720 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.720 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.098379_467.1", 2020-11-19T03:06:20.721 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.721 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:20.721 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.098379164Z", 2020-11-19T03:06:20.721 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.721 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.722 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:20.722 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.722 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.722 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:20.723 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.723 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.723 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.723 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.106961_468.1", 2020-11-19T03:06:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.106961657Z", 2020-11-19T03:06:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.187937_469.1", 2020-11-19T03:06:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36", 2020-11-19T03:06:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.187937642Z", 2020-11-19T03:06:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jRizeyH8HQBihpZqQ7H0cWx", 2020-11-19T03:06:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.191921_470.1", 2020-11-19T03:06:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36", 2020-11-19T03:06:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.191921477Z", 2020-11-19T03:06:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jRizeyH8HQBihpZqQ7H0cWx", 2020-11-19T03:06:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.735 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.735 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.736 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.193772_471.1", 2020-11-19T03:06:20.736 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.736 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-36", 2020-11-19T03:06:20.736 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.193772231Z", 2020-11-19T03:06:20.737 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.737 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.737 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.737 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.738 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.738 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_kYTvwOB6WrelYhgfkJRvPMC", 2020-11-19T03:06:20.738 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.738 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.739 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.739 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.739 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.740 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.197598_472.1", 2020-11-19T03:06:20.740 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.740 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-36", 2020-11-19T03:06:20.740 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.197598901Z", 2020-11-19T03:06:20.740 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.741 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.741 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.741 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.741 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.742 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_kYTvwOB6WrelYhgfkJRvPMC", 2020-11-19T03:06:20.742 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.742 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.742 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.743 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.743 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.743 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.235086_473.1", 2020-11-19T03:06:20.744 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.744 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37", 2020-11-19T03:06:20.744 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.235086410Z", 2020-11-19T03:06:20.744 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.744 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.745 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.745 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.745 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.745 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_my5SPqbG7dQQSEBcU42t2-N", 2020-11-19T03:06:20.746 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.746 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.746 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.746 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.747 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.747 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.239751_474.1", 2020-11-19T03:06:20.747 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.748 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37", 2020-11-19T03:06:20.748 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.239751394Z", 2020-11-19T03:06:20.748 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.748 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.748 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.749 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.749 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.749 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_my5SPqbG7dQQSEBcU42t2-N", 2020-11-19T03:06:20.749 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.750 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.750 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.750 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.750 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.751 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.242234_475.1", 2020-11-19T03:06:20.751 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.751 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-37", 2020-11-19T03:06:20.751 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.242234716Z", 2020-11-19T03:06:20.752 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.752 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.752 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.752 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.753 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.753 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_g91x9CrlSKmRRavm4MwY2r5", 2020-11-19T03:06:20.753 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.753 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.754 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.754 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.754 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.755 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.247176_476.1", 2020-11-19T03:06:20.755 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.755 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-37", 2020-11-19T03:06:20.755 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.247176520Z", 2020-11-19T03:06:20.755 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.756 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.756 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.756 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.756 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_g91x9CrlSKmRRavm4MwY2r5", 2020-11-19T03:06:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755038.595496_477.1", 2020-11-19T03:06:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38", 2020-11-19T03:06:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:58.595496690Z", 2020-11-19T03:06:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V7n32juggP5QmeLeOGxy3UK", 2020-11-19T03:06:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755038.599524_478.1", 2020-11-19T03:06:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38", 2020-11-19T03:06:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:58.599524203Z", 2020-11-19T03:06:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V7n32juggP5QmeLeOGxy3UK", 2020-11-19T03:06:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755038.601388_479.1", 2020-11-19T03:06:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-38", 2020-11-19T03:06:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:58.601388404Z", 2020-11-19T03:06:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_W1ZCfu17NUxLx7oS231exeH", 2020-11-19T03:06:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755038.605369_480.1", 2020-11-19T03:06:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-38", 2020-11-19T03:06:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:58.605369575Z", 2020-11-19T03:06:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_W1ZCfu17NUxLx7oS231exeH", 2020-11-19T03:06:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.318171_485.1", 2020-11-19T03:06:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-39:d71482d6-15fd-4715-9992-a25958dcb538.4341.39", 2020-11-19T03:06:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.318171704Z", 2020-11-19T03:06:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MR6DDMoW-bWzeQ2lVYjkv_T", 2020-11-19T03:06:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.326000_486.1", 2020-11-19T03:06:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-39:d71482d6-15fd-4715-9992-a25958dcb538.4341.39", 2020-11-19T03:06:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.326000603Z", 2020-11-19T03:06:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MR6DDMoW-bWzeQ2lVYjkv_T", 2020-11-19T03:06:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.328429_487.1", 2020-11-19T03:06:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-39", 2020-11-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.328429735Z", 2020-11-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2xs16O_Y3iMqtaOm0GzhjDK", 2020-11-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.333402_488.1", 2020-11-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-39", 2020-11-19T03:06:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.333402182Z", 2020-11-19T03:06:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2xs16O_Y3iMqtaOm0GzhjDK", 2020-11-19T03:06:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.366052_489.1", 2020-11-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-40:d71482d6-15fd-4715-9992-a25958dcb538.4341.40", 2020-11-19T03:06:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.366052352Z", 2020-11-19T03:06:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_oGE_O83RX1iSJnFe2p4H0J-", 2020-11-19T03:06:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.372774_490.1", 2020-11-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-40:d71482d6-15fd-4715-9992-a25958dcb538.4341.40", 2020-11-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.372774679Z", 2020-11-19T03:06:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_oGE_O83RX1iSJnFe2p4H0J-", 2020-11-19T03:06:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.375737_491.1", 2020-11-19T03:06:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-40", 2020-11-19T03:06:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.375737830Z", 2020-11-19T03:06:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bGn8OsdsLasWP-8EeCEuz7n", 2020-11-19T03:06:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.382371_492.1", 2020-11-19T03:06:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-40", 2020-11-19T03:06:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.382371633Z", 2020-11-19T03:06:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bGn8OsdsLasWP-8EeCEuz7n", 2020-11-19T03:06:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.429548_493.1", 2020-11-19T03:06:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-41:d71482d6-15fd-4715-9992-a25958dcb538.4341.41", 2020-11-19T03:06:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.429548543Z", 2020-11-19T03:06:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__wSJMtQIou6xYhPPtSS8YuA", 2020-11-19T03:06:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.436987_494.1", 2020-11-19T03:06:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-41:d71482d6-15fd-4715-9992-a25958dcb538.4341.41", 2020-11-19T03:06:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.436987958Z", 2020-11-19T03:06:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__wSJMtQIou6xYhPPtSS8YuA", 2020-11-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.439804_495.1", 2020-11-19T03:06:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-41", 2020-11-19T03:06:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.439804366Z", 2020-11-19T03:06:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_RsdLOBWehIFSw4jrAQphm1V", 2020-11-19T03:06:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.443579_496.1", 2020-11-19T03:06:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-41", 2020-11-19T03:06:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.443579496Z", 2020-11-19T03:06:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_RsdLOBWehIFSw4jrAQphm1V", 2020-11-19T03:06:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.049424_71.1", 2020-11-19T03:06:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-1:d71482d6-15fd-4715-9992-a25958dcb538.4341.1", 2020-11-19T03:06:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.049424391Z", 2020-11-19T03:06:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qgItci47XzVPPEUbRm9FP71", 2020-11-19T03:06:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.067007_75.1", 2020-11-19T03:06:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-1:d71482d6-15fd-4715-9992-a25958dcb538.4341.1", 2020-11-19T03:06:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.067007384Z", 2020-11-19T03:06:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qgItci47XzVPPEUbRm9FP71", 2020-11-19T03:06:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.068631_76.1", 2020-11-19T03:06:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-1", 2020-11-19T03:06:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.068631964Z", 2020-11-19T03:06:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_94mQ_WxhbwDk-kFSjKqwmYF", 2020-11-19T03:06:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.071728_77.1", 2020-11-19T03:06:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-1", 2020-11-19T03:06:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.071728952Z", 2020-11-19T03:06:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_94mQ_WxhbwDk-kFSjKqwmYF", 2020-11-19T03:06:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.094013_78.1", 2020-11-19T03:06:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-2:d71482d6-15fd-4715-9992-a25958dcb538.4341.2", 2020-11-19T03:06:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.094013583Z", 2020-11-19T03:06:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bSYBD3JTM-OkT97a99BJ1cL", 2020-11-19T03:06:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.098511_79.1", 2020-11-19T03:06:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-2:d71482d6-15fd-4715-9992-a25958dcb538.4341.2", 2020-11-19T03:06:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.098511431Z", 2020-11-19T03:06:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bSYBD3JTM-OkT97a99BJ1cL", 2020-11-19T03:06:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.100851_80.1", 2020-11-19T03:06:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-2", 2020-11-19T03:06:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.100851568Z", 2020-11-19T03:06:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.845 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.845 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.845 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.845 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5UuoR0SlRK-wajXJjy8RaHm", 2020-11-19T03:06:20.846 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.846 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.846 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.847 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.847 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.847 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.105586_81.1", 2020-11-19T03:06:20.848 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.848 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-2", 2020-11-19T03:06:20.848 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.105586168Z", 2020-11-19T03:06:20.848 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.848 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.849 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.849 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.849 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.849 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5UuoR0SlRK-wajXJjy8RaHm", 2020-11-19T03:06:20.850 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.850 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.850 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.850 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.851 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.851 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754538.238498_82.1", 2020-11-19T03:06:20.851 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.852 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-3:d71482d6-15fd-4715-9992-a25958dcb538.4341.3", 2020-11-19T03:06:20.852 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:38.238498971Z", 2020-11-19T03:06:20.852 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.852 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.852 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.853 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.853 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.853 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_GAW34u3sdJwjuChdK6wHKmm", 2020-11-19T03:06:20.853 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.854 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.854 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.854 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.854 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.855 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754538.263102_83.1", 2020-11-19T03:06:20.855 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.855 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-3:d71482d6-15fd-4715-9992-a25958dcb538.4341.3", 2020-11-19T03:06:20.856 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:38.263102074Z", 2020-11-19T03:06:20.856 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.856 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.856 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.856 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.857 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.857 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_GAW34u3sdJwjuChdK6wHKmm", 2020-11-19T03:06:20.857 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.858 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.858 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.858 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.858 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.859 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754538.265151_84.1", 2020-11-19T03:06:20.859 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.859 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-3", 2020-11-19T03:06:20.859 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:38.265151277Z", 2020-11-19T03:06:20.860 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.860 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.860 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.860 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.861 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.861 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_tL7nc-Jhsvjv2EEZRXT73OW", 2020-11-19T03:06:20.861 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.861 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.862 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.863 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754538.270230_85.1", 2020-11-19T03:06:20.863 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.863 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-3", 2020-11-19T03:06:20.863 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:38.270230968Z", 2020-11-19T03:06:20.863 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.864 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.864 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.864 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.864 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.865 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_tL7nc-Jhsvjv2EEZRXT73OW", 2020-11-19T03:06:20.865 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.865 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.865 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.866 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.866 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.866 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.646238_86.1", 2020-11-19T03:06:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-4:d71482d6-15fd-4715-9992-a25958dcb538.4341.4", 2020-11-19T03:06:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.646238073Z", 2020-11-19T03:06:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_kXyu9LGpqsO6w8QSZAhud3g", 2020-11-19T03:06:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.653311_88.1", 2020-11-19T03:06:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-4:d71482d6-15fd-4715-9992-a25958dcb538.4341.4", 2020-11-19T03:06:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.653311568Z", 2020-11-19T03:06:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_kXyu9LGpqsO6w8QSZAhud3g", 2020-11-19T03:06:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.874 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.655601_89.1", 2020-11-19T03:06:20.874 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.874 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-4", 2020-11-19T03:06:20.875 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.655601491Z", 2020-11-19T03:06:20.875 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.875 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.875 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.875 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.876 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.876 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_tMmjLsHDWqdVdqcqT0YXzke", 2020-11-19T03:06:20.876 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.877 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.877 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.877 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.877 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.878 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.661306_90.1", 2020-11-19T03:06:20.878 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.878 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-4", 2020-11-19T03:06:20.878 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.661306950Z", 2020-11-19T03:06:20.879 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.879 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.879 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.879 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.880 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.880 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_tMmjLsHDWqdVdqcqT0YXzke", 2020-11-19T03:06:20.880 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.880 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.881 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.881 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.881 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.882 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.685988_91.1", 2020-11-19T03:06:20.882 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.882 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-5:d71482d6-15fd-4715-9992-a25958dcb538.4341.5", 2020-11-19T03:06:20.882 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.685988313Z", 2020-11-19T03:06:20.882 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.883 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.883 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.883 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.883 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.884 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qVWvmOqSOj4nJ_IyDQjqQEP", 2020-11-19T03:06:20.884 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.884 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.884 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.885 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.885 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.886 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.690669_92.1", 2020-11-19T03:06:20.886 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.886 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-5:d71482d6-15fd-4715-9992-a25958dcb538.4341.5", 2020-11-19T03:06:20.886 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.690669131Z", 2020-11-19T03:06:20.886 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.887 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.887 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.887 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.887 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.888 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qVWvmOqSOj4nJ_IyDQjqQEP", 2020-11-19T03:06:20.888 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.888 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.888 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.889 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.889 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.889 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.693071_93.1", 2020-11-19T03:06:20.890 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.890 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-5", 2020-11-19T03:06:20.890 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.693071890Z", 2020-11-19T03:06:20.890 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.890 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.891 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.891 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.891 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.891 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_IRRwWfE5ELTX6B9PeD0Wss2", 2020-11-19T03:06:20.892 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.892 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.892 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.892 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.893 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.893 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.697214_94.1", 2020-11-19T03:06:20.893 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.894 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-5", 2020-11-19T03:06:20.894 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.697214643Z", 2020-11-19T03:06:20.894 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.894 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.894 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.895 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.895 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.895 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_IRRwWfE5ELTX6B9PeD0Wss2", 2020-11-19T03:06:20.895 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.896 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.896 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.896 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.896 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.897 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.740644_95.1", 2020-11-19T03:06:20.897 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.897 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-6:d71482d6-15fd-4715-9992-a25958dcb538.4341.6", 2020-11-19T03:06:20.898 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.740644196Z", 2020-11-19T03:06:20.898 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.898 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.898 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.898 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.899 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.899 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qrn-62ONHSx8esDlsD49N2S", 2020-11-19T03:06:20.899 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.899 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.900 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.900 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.900 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.901 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.744532_96.1", 2020-11-19T03:06:20.901 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.901 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-6:d71482d6-15fd-4715-9992-a25958dcb538.4341.6", 2020-11-19T03:06:20.901 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.744532390Z", 2020-11-19T03:06:20.901 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.902 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.902 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.902 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.902 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.903 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qrn-62ONHSx8esDlsD49N2S", 2020-11-19T03:06:20.903 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.903 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.903 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.904 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.904 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.904 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.746440_97.1", 2020-11-19T03:06:20.905 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.905 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-6", 2020-11-19T03:06:20.905 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.746440476Z", 2020-11-19T03:06:20.905 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.906 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.906 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.906 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.906 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.907 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_wFarfv5IWniL3C-MOrXA7X0", 2020-11-19T03:06:20.907 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.907 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.907 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.907 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.908 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.908 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.750854_98.1", 2020-11-19T03:06:20.909 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.909 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-6", 2020-11-19T03:06:20.909 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.750854665Z", 2020-11-19T03:06:20.909 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.909 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.910 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.910 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.910 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.910 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_wFarfv5IWniL3C-MOrXA7X0", 2020-11-19T03:06:20.911 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.911 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.911 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.911 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.912 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.912 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.678670_99.1", 2020-11-19T03:06:20.912 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.912 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7:d71482d6-15fd-4715-9992-a25958dcb538.4341.7", 2020-11-19T03:06:20.913 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.678670320Z", 2020-11-19T03:06:20.913 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.913 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.913 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.914 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.914 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.914 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3L9q5SfSgUsZej9VYTkemdm", 2020-11-19T03:06:20.914 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.915 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.915 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.915 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.915 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.916 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.684139_102.1", 2020-11-19T03:06:20.916 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.916 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7:d71482d6-15fd-4715-9992-a25958dcb538.4341.7", 2020-11-19T03:06:20.916 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.684139704Z", 2020-11-19T03:06:20.917 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.917 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.917 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.917 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.918 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.918 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3L9q5SfSgUsZej9VYTkemdm", 2020-11-19T03:06:20.918 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.918 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.919 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.919 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.919 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.920 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.686054_103.1", 2020-11-19T03:06:20.920 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.920 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7", 2020-11-19T03:06:20.920 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.686054368Z", 2020-11-19T03:06:20.920 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.921 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.921 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.921 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.921 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.922 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:20.922 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.922 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.922 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.923 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.690172_104.1", 2020-11-19T03:06:20.924 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.924 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7", 2020-11-19T03:06:20.924 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.690172550Z", 2020-11-19T03:06:20.924 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.924 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.925 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.925 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.925 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.925 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:20.926 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.926 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.926 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.927 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.714743_105.1", 2020-11-19T03:06:20.928 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.928 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8:d71482d6-15fd-4715-9992-a25958dcb538.4341.8", 2020-11-19T03:06:20.928 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.714743025Z", 2020-11-19T03:06:20.928 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.928 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.929 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.929 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.929 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.929 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Y2tOicXm39NSETUiNVpNZLc", 2020-11-19T03:06:20.930 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.930 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.930 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.931 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.931 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.719653_106.1", 2020-11-19T03:06:20.931 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.932 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8:d71482d6-15fd-4715-9992-a25958dcb538.4341.8", 2020-11-19T03:06:20.932 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.719653546Z", 2020-11-19T03:06:20.932 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.932 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.932 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.933 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.933 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.933 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Y2tOicXm39NSETUiNVpNZLc", 2020-11-19T03:06:20.933 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.934 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.934 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.934 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.934 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.935 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.722344_107.1", 2020-11-19T03:06:20.935 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.935 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8", 2020-11-19T03:06:20.936 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.722344069Z", 2020-11-19T03:06:20.936 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.936 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.936 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.936 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.937 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.937 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:20.937 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.938 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.938 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.939 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.727853_108.1", 2020-11-19T03:06:20.939 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.939 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8", 2020-11-19T03:06:20.939 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.727853163Z", 2020-11-19T03:06:20.940 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.940 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.940 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.940 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.941 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.941 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:20.941 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.942 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.942 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.942 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.942 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.943 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.773887_109.1", 2020-11-19T03:06:20.943 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.943 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9:d71482d6-15fd-4715-9992-a25958dcb538.4341.9", 2020-11-19T03:06:20.943 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.773887989Z", 2020-11-19T03:06:20.943 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.944 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.944 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.944 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.944 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.945 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_vGJjA-wwP6rD2bH8RFgbNPS", 2020-11-19T03:06:20.945 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.945 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.945 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.946 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.946 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.946 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.778690_110.1", 2020-11-19T03:06:20.947 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:20.947 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9:d71482d6-15fd-4715-9992-a25958dcb538.4341.9", 2020-11-19T03:06:20.947 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.778690126Z", 2020-11-19T03:06:20.947 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.948 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.948 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.948 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.948 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.949 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_vGJjA-wwP6rD2bH8RFgbNPS", 2020-11-19T03:06:20.949 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.949 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.950 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.950 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.950 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.951 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.781507_111.1", 2020-11-19T03:06:20.951 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.951 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9", 2020-11-19T03:06:20.951 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.781507411Z", 2020-11-19T03:06:20.951 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.951 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.952 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.952 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.952 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.952 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:20.953 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.953 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.953 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:20.953 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.954 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.954 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.787262_112.1", 2020-11-19T03:06:20.954 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.955 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9", 2020-11-19T03:06:20.955 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.787262520Z", 2020-11-19T03:06:20.955 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.955 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.955 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:20.956 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:20.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.956 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.956 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:20.957 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.957 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.957 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.957 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.958 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.958 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.124200_113.1", 2020-11-19T03:06:20.958 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.959 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7", 2020-11-19T03:06:20.959 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.124200729Z", 2020-11-19T03:06:20.959 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.959 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.959 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:20.960 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.960 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.960 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:20.960 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.961 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.961 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "remove" 2020-11-19T03:06:20.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.962 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.128855_115.1", 2020-11-19T03:06:20.962 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.962 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7", 2020-11-19T03:06:20.962 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.128855513Z", 2020-11-19T03:06:20.963 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.963 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.963 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:20.963 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.964 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.964 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:20.964 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.964 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.965 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.965 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.965 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.966 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.183608_116.1", 2020-11-19T03:06:20.966 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.966 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8", 2020-11-19T03:06:20.966 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.183608180Z", 2020-11-19T03:06:20.966 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.967 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.967 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:20.967 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.967 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.968 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:20.968 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.968 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.968 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "remove" 2020-11-19T03:06:20.969 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.969 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.969 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.189052_117.1", 2020-11-19T03:06:20.970 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.970 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8", 2020-11-19T03:06:20.970 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.189052423Z", 2020-11-19T03:06:20.970 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.970 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.971 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:20.971 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.971 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.971 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:20.972 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.972 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.972 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.973 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.973 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.288556_118.1", 2020-11-19T03:06:20.973 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.974 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9", 2020-11-19T03:06:20.974 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.288556017Z", 2020-11-19T03:06:20.974 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.974 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.974 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:20.975 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.975 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.975 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:20.975 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.976 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.976 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "remove" 2020-11-19T03:06:20.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:20.977 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.294580_119.1", 2020-11-19T03:06:20.977 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:20.977 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9", 2020-11-19T03:06:20.978 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580366Z", 2020-11-19T03:06:20.978 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:20.978 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:20.978 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:20.978 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:20.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.979 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:20.979 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:20.979 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:20.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:20.980 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:20.980 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:20.980 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.980 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.980 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:20.981 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:20.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:20.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:20.983 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b2c8afed-318c-484c-8dd4-daab7a757391 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:21.026 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:21.027 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:21.274 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:06:21.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.274 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754414.632290_106.1", 2020-11-19T03:06:21.275 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "user", 2020-11-19T03:06:21.275 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "rgw-multisite-test-user", 2020-11-19T03:06:21.276 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.632290363Z", 2020-11-19T03:06:21.276 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.276 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.277 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.277 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.278 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.279 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OD787OEqCb37r-dWrIu2W6g", 2020-11-19T03:06:21.279 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.280 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.280 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.282 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754414.640760_109.1", 2020-11-19T03:06:21.282 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "user", 2020-11-19T03:06:21.282 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "rgw-multisite-test-user", 2020-11-19T03:06:21.283 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760721Z", 2020-11-19T03:06:21.283 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.283 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.283 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.284 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.284 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.284 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OD787OEqCb37r-dWrIu2W6g", 2020-11-19T03:06:21.284 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.285 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.285 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.286 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.755439_114.1", 2020-11-19T03:06:21.286 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.287 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10", 2020-11-19T03:06:21.287 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.755439437Z", 2020-11-19T03:06:21.287 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.287 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.287 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.288 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.288 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.288 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jDQMSP1imZk9M53g_LemGsD", 2020-11-19T03:06:21.288 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.289 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.289 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.290 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.291 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.759756_127.1", 2020-11-19T03:06:21.291 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.291 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10", 2020-11-19T03:06:21.292 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.759756660Z", 2020-11-19T03:06:21.292 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.294 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.295 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jDQMSP1imZk9M53g_LemGsD", 2020-11-19T03:06:21.295 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.296 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.296 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.298 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.761697_128.1", 2020-11-19T03:06:21.299 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.299 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-10", 2020-11-19T03:06:21.300 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.761697955Z", 2020-11-19T03:06:21.300 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.300 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.301 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.301 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.302 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.302 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_QaR_8xY7tB3j-5_fwfgIAp2", 2020-11-19T03:06:21.303 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.304 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.304 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.306 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.765274_129.1", 2020-11-19T03:06:21.306 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.306 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-10", 2020-11-19T03:06:21.306 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.765274847Z", 2020-11-19T03:06:21.306 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.307 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.307 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.307 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.307 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.308 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_QaR_8xY7tB3j-5_fwfgIAp2", 2020-11-19T03:06:21.308 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.308 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.308 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.309 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.309 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.310 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.789431_130.1", 2020-11-19T03:06:21.310 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.310 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11", 2020-11-19T03:06:21.310 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.789431860Z", 2020-11-19T03:06:21.310 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.311 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.311 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.311 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.311 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.312 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_h4zkJ0TD5OOIK8n5LNF_3YA", 2020-11-19T03:06:21.312 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.312 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.312 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.313 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.793774_131.1", 2020-11-19T03:06:21.314 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.314 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11", 2020-11-19T03:06:21.314 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.793774959Z", 2020-11-19T03:06:21.314 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.314 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.315 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.315 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.315 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.315 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_h4zkJ0TD5OOIK8n5LNF_3YA", 2020-11-19T03:06:21.316 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.316 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.316 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.316 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.317 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.317 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.798353_132.1", 2020-11-19T03:06:21.317 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.318 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-11", 2020-11-19T03:06:21.318 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.798353113Z", 2020-11-19T03:06:21.318 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.318 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.318 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.319 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.319 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.319 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5lbvhy-2ssHalMo0XAyb9jK", 2020-11-19T03:06:21.319 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.320 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.320 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.321 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.803735_133.1", 2020-11-19T03:06:21.321 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.321 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-11", 2020-11-19T03:06:21.322 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.803735217Z", 2020-11-19T03:06:21.322 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.322 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.322 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.322 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5lbvhy-2ssHalMo0XAyb9jK", 2020-11-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.854454_134.1", 2020-11-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12", 2020-11-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.854454685Z", 2020-11-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.327 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.327 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_DG8ZKomIxnXa3vMXcvP8ajO", 2020-11-19T03:06:21.327 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.327 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.328 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.329 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.858973_135.1", 2020-11-19T03:06:21.329 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.329 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12", 2020-11-19T03:06:21.329 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.858973052Z", 2020-11-19T03:06:21.329 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.330 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.330 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.330 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.330 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.331 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_DG8ZKomIxnXa3vMXcvP8ajO", 2020-11-19T03:06:21.331 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.331 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.331 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.332 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.332 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.332 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.860848_136.1", 2020-11-19T03:06:21.333 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.333 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-12", 2020-11-19T03:06:21.333 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.860848274Z", 2020-11-19T03:06:21.333 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.334 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.334 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.334 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.334 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.335 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xHh-uu7OMNHE3E1tkdnCoN0", 2020-11-19T03:06:21.335 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.335 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.335 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.336 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.336 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.336 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.866167_137.1", 2020-11-19T03:06:21.336 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.337 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-12", 2020-11-19T03:06:21.337 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.866167085Z", 2020-11-19T03:06:21.337 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.337 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.337 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.338 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.338 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.338 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xHh-uu7OMNHE3E1tkdnCoN0", 2020-11-19T03:06:21.338 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.339 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.339 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.340 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.787642_145.1", 2020-11-19T03:06:21.340 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.340 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13", 2020-11-19T03:06:21.341 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.787642625Z", 2020-11-19T03:06:21.341 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.341 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.341 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.341 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.342 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.342 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qxTzHkbF9wemeZVynRNqeCj", 2020-11-19T03:06:21.342 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.343 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.343 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.344 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.792198_150.1", 2020-11-19T03:06:21.344 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.344 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13", 2020-11-19T03:06:21.344 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.792198045Z", 2020-11-19T03:06:21.345 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.345 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.345 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.345 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.346 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.346 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qxTzHkbF9wemeZVynRNqeCj", 2020-11-19T03:06:21.346 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.346 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.347 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.347 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.347 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.348 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.794166_151.1", 2020-11-19T03:06:21.348 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.348 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-13", 2020-11-19T03:06:21.348 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.794166672Z", 2020-11-19T03:06:21.348 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.349 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.349 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.349 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.349 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.350 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ogTrNysaHhYexwepXbm3dwI", 2020-11-19T03:06:21.350 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.350 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.350 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.351 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.351 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.352 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.798068_152.1", 2020-11-19T03:06:21.352 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.352 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-13", 2020-11-19T03:06:21.352 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.798068026Z", 2020-11-19T03:06:21.352 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.353 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.353 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.353 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.353 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.354 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ogTrNysaHhYexwepXbm3dwI", 2020-11-19T03:06:21.354 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.354 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.354 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.355 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.355 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.355 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.819538_154.1", 2020-11-19T03:06:21.356 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.356 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14", 2020-11-19T03:06:21.356 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.819538205Z", 2020-11-19T03:06:21.356 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.356 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.356 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.357 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.357 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.357 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dcIW3-alzbWV6sgq13MRCG7", 2020-11-19T03:06:21.358 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.358 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.358 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.358 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.359 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.359 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.825038_155.1", 2020-11-19T03:06:21.359 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.359 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14", 2020-11-19T03:06:21.360 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.825038088Z", 2020-11-19T03:06:21.360 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.360 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.360 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.360 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.361 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.361 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dcIW3-alzbWV6sgq13MRCG7", 2020-11-19T03:06:21.361 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.362 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.362 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.362 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.362 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.363 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.827341_156.1", 2020-11-19T03:06:21.363 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.364 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-14", 2020-11-19T03:06:21.364 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.827341897Z", 2020-11-19T03:06:21.364 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.364 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.364 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.365 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.365 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.365 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_TbVBTXwBuNJEI3zmckRRIjz", 2020-11-19T03:06:21.366 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.366 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.366 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.367 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.832699_157.1", 2020-11-19T03:06:21.368 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.368 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-14", 2020-11-19T03:06:21.368 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.832699383Z", 2020-11-19T03:06:21.368 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.368 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.369 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.369 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.369 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.369 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_TbVBTXwBuNJEI3zmckRRIjz", 2020-11-19T03:06:21.370 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.370 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.370 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.370 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.371 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.371 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.868186_158.1", 2020-11-19T03:06:21.371 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.372 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15", 2020-11-19T03:06:21.372 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.868186115Z", 2020-11-19T03:06:21.372 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.372 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.372 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.373 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.373 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.373 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Pzb_ZFMDCfzlSj1lsKyhj_z", 2020-11-19T03:06:21.373 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.374 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.374 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.375 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.872546_159.1", 2020-11-19T03:06:21.375 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.375 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15", 2020-11-19T03:06:21.376 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.872546446Z", 2020-11-19T03:06:21.376 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.376 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.376 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.376 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.377 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.377 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Pzb_ZFMDCfzlSj1lsKyhj_z", 2020-11-19T03:06:21.377 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.378 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.378 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.378 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.378 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.379 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.876242_160.1", 2020-11-19T03:06:21.379 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.379 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-15", 2020-11-19T03:06:21.380 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.876242156Z", 2020-11-19T03:06:21.380 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.380 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.380 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.380 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.381 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.381 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Hkg8dq-5IxwuJ-1F2ioM3Ha", 2020-11-19T03:06:21.381 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.382 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.382 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.382 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.382 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.383 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.888825_161.1", 2020-11-19T03:06:21.383 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.383 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-15", 2020-11-19T03:06:21.383 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.888825992Z", 2020-11-19T03:06:21.384 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.384 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.384 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.384 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.385 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.385 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Hkg8dq-5IxwuJ-1F2ioM3Ha", 2020-11-19T03:06:21.385 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.385 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.386 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.387 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.798543_207.1", 2020-11-19T03:06:21.387 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.387 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:21.387 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.798543679Z", 2020-11-19T03:06:21.387 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.388 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.388 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.388 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.388 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.389 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:21.389 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.389 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.389 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.390 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.803204_211.1", 2020-11-19T03:06:21.391 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.391 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:21.391 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.803204912Z", 2020-11-19T03:06:21.391 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.391 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.392 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.392 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.392 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.392 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:21.393 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.393 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.393 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.393 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.394 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.394 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.805294_212.1", 2020-11-19T03:06:21.394 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.395 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16", 2020-11-19T03:06:21.395 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.805294677Z", 2020-11-19T03:06:21.395 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.395 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.395 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.396 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.396 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.396 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8SdF_vsimBqHGDaqiMgkarR", 2020-11-19T03:06:21.396 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.397 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.397 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.397 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.397 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.398 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.810747_213.1", 2020-11-19T03:06:21.398 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.398 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16", 2020-11-19T03:06:21.398 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.810747502Z", 2020-11-19T03:06:21.399 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.399 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.399 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.399 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.400 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.400 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8SdF_vsimBqHGDaqiMgkarR", 2020-11-19T03:06:21.400 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.400 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.401 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.401 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.401 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.402 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.838151_214.1", 2020-11-19T03:06:21.402 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.402 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:21.402 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.838151294Z", 2020-11-19T03:06:21.402 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.403 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.403 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.403 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.403 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.404 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:21.404 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.404 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.405 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.406 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.842707_215.1", 2020-11-19T03:06:21.406 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.406 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:21.406 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.842707380Z", 2020-11-19T03:06:21.406 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.407 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.407 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.407 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.407 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.408 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:21.408 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.408 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.408 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.409 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.409 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.409 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.844853_216.1", 2020-11-19T03:06:21.410 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.410 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17", 2020-11-19T03:06:21.410 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.844853013Z", 2020-11-19T03:06:21.410 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.410 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.411 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.411 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.411 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.411 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__nKT7kdBFED-s1dk1lgSXVH", 2020-11-19T03:06:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.413 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.413 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.849738_217.1", 2020-11-19T03:06:21.414 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.414 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17", 2020-11-19T03:06:21.414 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.849738526Z", 2020-11-19T03:06:21.414 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.414 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.415 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.415 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.415 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.415 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__nKT7kdBFED-s1dk1lgSXVH", 2020-11-19T03:06:21.416 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.416 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.416 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.416 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.417 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.417 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.905621_218.1", 2020-11-19T03:06:21.417 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.418 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:21.418 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.905621496Z", 2020-11-19T03:06:21.418 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.418 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.418 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.419 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.419 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.419 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:21.419 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.420 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.420 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.420 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.420 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.911693_219.1", 2020-11-19T03:06:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:21.422 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.911693551Z", 2020-11-19T03:06:21.422 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.422 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.422 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.422 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.423 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.423 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:21.423 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.424 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.424 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.424 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.424 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.425 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.915252_220.1", 2020-11-19T03:06:21.425 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.425 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18", 2020-11-19T03:06:21.426 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.915252896Z", 2020-11-19T03:06:21.426 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.426 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.426 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.426 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.427 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.427 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6l8yn5mDyDSfq7DcU_4ezuR", 2020-11-19T03:06:21.427 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.427 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.428 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.428 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.428 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.429 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.919671_221.1", 2020-11-19T03:06:21.429 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.429 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18", 2020-11-19T03:06:21.430 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.919671168Z", 2020-11-19T03:06:21.430 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.430 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.430 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.430 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.431 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.431 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6l8yn5mDyDSfq7DcU_4ezuR", 2020-11-19T03:06:21.431 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.432 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.432 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.432 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.432 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.433 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.939873_222.1", 2020-11-19T03:06:21.433 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.433 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:21.435 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.939873846Z", 2020-11-19T03:06:21.435 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.436 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.436 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:21.436 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.436 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.437 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:21.437 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.437 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.437 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.438 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.438 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.438 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.948027_223.1", 2020-11-19T03:06:21.439 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.439 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:21.439 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.948027995Z", 2020-11-19T03:06:21.439 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.439 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.440 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:21.440 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.440 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.440 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:21.441 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.441 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.441 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.441 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.442 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.442 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.955102_224.1", 2020-11-19T03:06:21.442 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.443 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:21.443 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.955102556Z", 2020-11-19T03:06:21.443 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.443 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.443 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:21.444 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.444 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.444 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:21.444 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.445 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.445 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.445 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.446 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.446 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.962377_225.1", 2020-11-19T03:06:21.446 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.447 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:21.447 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.962377954Z", 2020-11-19T03:06:21.447 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.447 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.447 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:21.448 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.448 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.448 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:21.448 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.449 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.449 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.449 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.449 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.450 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.974309_226.1", 2020-11-19T03:06:21.450 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.450 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:21.451 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.974309142Z", 2020-11-19T03:06:21.451 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.451 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.451 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:21.451 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.452 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.452 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:21.452 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.453 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.453 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.453 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.453 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.454 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.983290_227.1", 2020-11-19T03:06:21.454 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.454 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:21.454 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.983290859Z", 2020-11-19T03:06:21.455 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.455 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.455 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:21.455 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.456 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.456 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:21.456 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.456 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.457 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.457 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.457 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.458 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.853615_253.1", 2020-11-19T03:06:21.458 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.458 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:21.458 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.853615363Z", 2020-11-19T03:06:21.458 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.459 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.459 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.459 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.459 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.460 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:21.460 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.460 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.460 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.461 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.461 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.461 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.857577_262.1", 2020-11-19T03:06:21.462 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.462 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:21.462 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.857577608Z", 2020-11-19T03:06:21.462 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.462 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.463 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:21.464 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.464 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.464 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.859745_263.1", 2020-11-19T03:06:21.465 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.466 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19", 2020-11-19T03:06:21.466 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.859745204Z", 2020-11-19T03:06:21.466 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.466 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.466 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Qy4UpA_MiVj8fLm1yoUPwmx", 2020-11-19T03:06:21.467 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.468 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.468 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.468 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.468 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.469 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.863407_264.1", 2020-11-19T03:06:21.469 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.470 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19", 2020-11-19T03:06:21.470 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.863407887Z", 2020-11-19T03:06:21.470 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.470 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.470 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Qy4UpA_MiVj8fLm1yoUPwmx", 2020-11-19T03:06:21.471 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.472 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.473 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.869018_265.1", 2020-11-19T03:06:21.473 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.473 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:21.474 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.869018103Z", 2020-11-19T03:06:21.474 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.474 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.474 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.476 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.476 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.476 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.476 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.477 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.874924_266.1", 2020-11-19T03:06:21.477 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.477 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:21.478 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.874924462Z", 2020-11-19T03:06:21.478 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.478 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.478 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:21.478 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.479 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.479 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:21.479 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.479 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.480 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.481 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754893.333355_282.1", 2020-11-19T03:06:21.481 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.481 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:21.481 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.333355116Z", 2020-11-19T03:06:21.481 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.482 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.482 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.482 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.482 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.483 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.483 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.483 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.483 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.484 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.484 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.484 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754893.338774_293.1", 2020-11-19T03:06:21.485 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.485 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:21.485 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.338774111Z", 2020-11-19T03:06:21.485 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.485 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.486 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.486 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.486 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.486 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.487 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.487 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.487 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.488 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.488 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754893.341581_294.1", 2020-11-19T03:06:21.488 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.489 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20", 2020-11-19T03:06:21.489 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.341581490Z", 2020-11-19T03:06:21.489 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.489 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.489 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.490 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.490 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.490 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_m4q0aNW1r6ONbHC6SDkHRNx", 2020-11-19T03:06:21.490 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.492 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754893.346605_295.1", 2020-11-19T03:06:21.492 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.492 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20", 2020-11-19T03:06:21.493 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.346605721Z", 2020-11-19T03:06:21.493 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.493 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.493 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.493 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.494 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.494 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_m4q0aNW1r6ONbHC6SDkHRNx", 2020-11-19T03:06:21.494 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.495 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.495 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.496 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754898.951630_299.1", 2020-11-19T03:06:21.496 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.496 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:21.496 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:38.951630338Z", 2020-11-19T03:06:21.497 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.497 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.497 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.497 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.498 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.498 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.498 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.498 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.499 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.500 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754898.962533_304.1", 2020-11-19T03:06:21.500 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.500 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:21.500 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:38.962533901Z", 2020-11-19T03:06:21.500 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.501 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.501 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.501 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.501 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.502 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.502 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.502 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.502 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.503 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.503 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.503 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754969.539612_314.1", 2020-11-19T03:06:21.504 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.504 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:21.504 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:02:49.539612219Z", 2020-11-19T03:06:21.504 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.504 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.505 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.505 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.505 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.505 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.506 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:21.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.506 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.506 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.506 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.507 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.507 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754969.549049_315.1", 2020-11-19T03:06:21.507 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.508 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:21.508 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:02:49.549049417Z", 2020-11-19T03:06:21.508 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.508 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.508 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.509 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.509 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.509 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:21.509 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:21.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.510 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.510 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.511 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.481687_339.1", 2020-11-19T03:06:21.511 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.511 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.481687459Z", 2020-11-19T03:06:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.486007_340.1", 2020-11-19T03:06:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.486007579Z", 2020-11-19T03:06:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.487864_341.1", 2020-11-19T03:06:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21", 2020-11-19T03:06:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.487864468Z", 2020-11-19T03:06:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UwTo_benAnYmaa4CegfHe2h", 2020-11-19T03:06:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.492471_342.1", 2020-11-19T03:06:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21", 2020-11-19T03:06:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.492471230Z", 2020-11-19T03:06:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UwTo_benAnYmaa4CegfHe2h", 2020-11-19T03:06:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.516426_343.1", 2020-11-19T03:06:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.516426936Z", 2020-11-19T03:06:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.520864_344.1", 2020-11-19T03:06:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.520864137Z", 2020-11-19T03:06:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.522824_345.1", 2020-11-19T03:06:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22", 2020-11-19T03:06:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.522824373Z", 2020-11-19T03:06:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-PCUgWQeHT2U7eFpqPZrnJx", 2020-11-19T03:06:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.527162_346.1", 2020-11-19T03:06:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22", 2020-11-19T03:06:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.527162988Z", 2020-11-19T03:06:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-PCUgWQeHT2U7eFpqPZrnJx", 2020-11-19T03:06:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.572422_347.1", 2020-11-19T03:06:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.572422879Z", 2020-11-19T03:06:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.578468_348.1", 2020-11-19T03:06:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.578468758Z", 2020-11-19T03:06:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.580555_349.1", 2020-11-19T03:06:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23", 2020-11-19T03:06:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.580555073Z", 2020-11-19T03:06:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_PtmtnLQARXBOkzZFBwIKFhM", 2020-11-19T03:06:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.585888_350.1", 2020-11-19T03:06:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23", 2020-11-19T03:06:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.585888316Z", 2020-11-19T03:06:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_PtmtnLQARXBOkzZFBwIKFhM", 2020-11-19T03:06:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.610407_351.1", 2020-11-19T03:06:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.610407033Z", 2020-11-19T03:06:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.616786_352.1", 2020-11-19T03:06:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.616786647Z", 2020-11-19T03:06:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.625234_353.1", 2020-11-19T03:06:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.625234108Z", 2020-11-19T03:06:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.633175_354.1", 2020-11-19T03:06:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.633175833Z", 2020-11-19T03:06:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.644998_355.1", 2020-11-19T03:06:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.644998413Z", 2020-11-19T03:06:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.660851_356.1", 2020-11-19T03:06:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.660851700Z", 2020-11-19T03:06:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.005483_361.1", 2020-11-19T03:06:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.005483628Z", 2020-11-19T03:06:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.009877_374.1", 2020-11-19T03:06:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.009877496Z", 2020-11-19T03:06:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.012023_375.1", 2020-11-19T03:06:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24", 2020-11-19T03:06:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.012023407Z", 2020-11-19T03:06:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ycH_LWaCb7pN9H3I9w2Z6N_", 2020-11-19T03:06:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.016947_376.1", 2020-11-19T03:06:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24", 2020-11-19T03:06:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.016947542Z", 2020-11-19T03:06:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ycH_LWaCb7pN9H3I9w2Z6N_", 2020-11-19T03:06:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.045357_377.1", 2020-11-19T03:06:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.045357346Z", 2020-11-19T03:06:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.050883_378.1", 2020-11-19T03:06:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.050883011Z", 2020-11-19T03:06:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.053418_379.1", 2020-11-19T03:06:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25", 2020-11-19T03:06:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.053418708Z", 2020-11-19T03:06:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bkpajY1jnfqrwJDp6NVto29", 2020-11-19T03:06:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.059396_380.1", 2020-11-19T03:06:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25", 2020-11-19T03:06:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.059396024Z", 2020-11-19T03:06:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bkpajY1jnfqrwJDp6NVto29", 2020-11-19T03:06:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.092006_381.1", 2020-11-19T03:06:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.092006149Z", 2020-11-19T03:06:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.096908_382.1", 2020-11-19T03:06:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.096908426Z", 2020-11-19T03:06:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.099132_383.1", 2020-11-19T03:06:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26", 2020-11-19T03:06:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.099132448Z", 2020-11-19T03:06:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.628 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.628 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_WAoNjTvGKPea_y2zejmRi9k", 2020-11-19T03:06:21.628 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.629 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.629 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.630 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.109731_384.1", 2020-11-19T03:06:21.630 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.630 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26", 2020-11-19T03:06:21.630 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.109731627Z", 2020-11-19T03:06:21.631 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.631 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.631 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.631 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.632 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.632 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_WAoNjTvGKPea_y2zejmRi9k", 2020-11-19T03:06:21.632 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.633 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.633 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.634 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.128366_385.1", 2020-11-19T03:06:21.634 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.634 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:21.634 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.128366050Z", 2020-11-19T03:06:21.635 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.635 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.635 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.635 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.636 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.636 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.636 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.636 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.637 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.637 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.637 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.638 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.134273_386.1", 2020-11-19T03:06:21.638 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.638 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:21.638 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.134273445Z", 2020-11-19T03:06:21.638 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.639 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.639 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.639 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.640 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.640 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.640 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.640 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.641 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.642 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.138369_387.1", 2020-11-19T03:06:21.642 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.642 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:21.642 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.138369812Z", 2020-11-19T03:06:21.642 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.643 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.643 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.643 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.643 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.644 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.644 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:21.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.644 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.644 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.645 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.151920_388.1", 2020-11-19T03:06:21.646 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.646 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:21.646 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.151920363Z", 2020-11-19T03:06:21.646 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.646 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.647 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.647 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.647 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.647 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:21.648 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:21.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.648 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.648 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.648 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.649 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.649 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.157598_389.1", 2020-11-19T03:06:21.649 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.650 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:21.650 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.157598827Z", 2020-11-19T03:06:21.650 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.650 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.650 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.651 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.651 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.651 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.651 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.652 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.652 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.652 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.653 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.653 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.167497_390.1", 2020-11-19T03:06:21.653 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.654 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:21.654 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.167497429Z", 2020-11-19T03:06:21.654 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.654 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.654 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.655 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.655 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.655 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.655 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.656 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.656 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.656 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.656 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.657 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.178487_391.1", 2020-11-19T03:06:21.657 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.657 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:21.658 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.178487831Z", 2020-11-19T03:06:21.658 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.658 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.658 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.658 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.659 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.659 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.659 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:21.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.660 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.660 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.660 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.660 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.661 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.186130_392.1", 2020-11-19T03:06:21.661 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.661 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:21.661 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.186130781Z", 2020-11-19T03:06:21.662 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.662 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.662 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.662 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.663 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.663 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:21.663 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:21.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.663 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.664 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.664 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.664 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.665 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.200136_393.1", 2020-11-19T03:06:21.665 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.665 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:21.665 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.200136493Z", 2020-11-19T03:06:21.666 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.666 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.666 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.666 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.667 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.667 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.667 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.667 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.668 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.668 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.668 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.669 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.228723_394.1", 2020-11-19T03:06:21.669 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.669 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:21.669 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.228723559Z", 2020-11-19T03:06:21.669 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.670 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.670 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.670 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.670 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.671 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.671 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.671 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.671 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.673 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.241089_395.1", 2020-11-19T03:06:21.673 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.673 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:21.673 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.241089363Z", 2020-11-19T03:06:21.673 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.674 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.674 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.674 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.674 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.675 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.675 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:21.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.675 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.675 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.676 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.248167_396.1", 2020-11-19T03:06:21.677 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.677 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:21.677 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.248167458Z", 2020-11-19T03:06:21.677 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.677 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.678 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.678 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.678 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.678 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:21.679 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:21.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.679 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.679 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.679 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.680 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.680 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.070651_403.1", 2020-11-19T03:06:21.680 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.681 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:21.681 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.070651958Z", 2020-11-19T03:06:21.681 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.681 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.681 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.682 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.682 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.682 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:21.682 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.683 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.683 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.683 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.683 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.684 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.075617_416.1", 2020-11-19T03:06:21.684 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.685 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:21.685 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.075617906Z", 2020-11-19T03:06:21.685 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.685 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.685 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.686 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.686 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.686 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:21.686 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.687 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.687 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.687 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.687 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.688 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.077978_417.1", 2020-11-19T03:06:21.688 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.688 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27", 2020-11-19T03:06:21.689 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.077978681Z", 2020-11-19T03:06:21.689 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.689 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.689 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.689 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.690 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.690 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jwhmoKKel5WXgN7QZadkAxM", 2020-11-19T03:06:21.690 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.691 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.691 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.692 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.081633_418.1", 2020-11-19T03:06:21.692 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.692 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27", 2020-11-19T03:06:21.692 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.081633830Z", 2020-11-19T03:06:21.693 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.693 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.693 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.693 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.694 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.694 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jwhmoKKel5WXgN7QZadkAxM", 2020-11-19T03:06:21.694 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.695 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.695 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.696 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.106122_419.1", 2020-11-19T03:06:21.696 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.696 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:21.696 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.106122712Z", 2020-11-19T03:06:21.697 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.697 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.697 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.697 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.698 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.698 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:21.698 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.699 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.699 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.699 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.699 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.700 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.111140_420.1", 2020-11-19T03:06:21.700 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.700 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:21.700 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.111140679Z", 2020-11-19T03:06:21.701 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.701 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.701 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.701 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.702 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.702 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:21.702 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.702 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.703 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.704 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.113526_421.1", 2020-11-19T03:06:21.704 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.704 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28", 2020-11-19T03:06:21.704 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.113526370Z", 2020-11-19T03:06:21.704 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.705 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.705 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.705 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.705 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.706 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_BALtCKJ9w_4jnJuGrJpKnG-", 2020-11-19T03:06:21.706 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.706 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.707 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.708 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.118961_422.1", 2020-11-19T03:06:21.708 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.708 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28", 2020-11-19T03:06:21.708 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.118961566Z", 2020-11-19T03:06:21.708 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.709 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.709 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.709 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.709 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.710 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_BALtCKJ9w_4jnJuGrJpKnG-", 2020-11-19T03:06:21.710 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.710 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.710 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.711 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.711 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.711 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.167348_423.1", 2020-11-19T03:06:21.712 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.712 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:21.712 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.167348445Z", 2020-11-19T03:06:21.712 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.712 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.713 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.713 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.713 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.714 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:21.714 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.714 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.714 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.714 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.715 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.715 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.172591_424.1", 2020-11-19T03:06:21.716 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.716 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:21.716 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.172591027Z", 2020-11-19T03:06:21.716 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.716 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.717 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:21.718 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.718 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.718 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.718 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.174666_425.1", 2020-11-19T03:06:21.719 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.720 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29", 2020-11-19T03:06:21.720 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.174666875Z", 2020-11-19T03:06:21.720 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.720 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.720 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7NmprpCcXUiFYkFKdzDwvBW", 2020-11-19T03:06:21.721 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.722 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.722 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.722 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.722 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.723 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.178503_426.1", 2020-11-19T03:06:21.723 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.723 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29", 2020-11-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.178503216Z", 2020-11-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.724 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.725 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.725 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7NmprpCcXUiFYkFKdzDwvBW", 2020-11-19T03:06:21.725 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.726 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.726 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.726 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.726 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.727 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.200545_427.1", 2020-11-19T03:06:21.727 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.727 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:21.728 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.200545045Z", 2020-11-19T03:06:21.728 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.728 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.728 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:21.728 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.729 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.729 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:21.729 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.730 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.730 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.730 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.730 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.731 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.208326_428.1", 2020-11-19T03:06:21.731 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.731 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:21.731 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.208326306Z", 2020-11-19T03:06:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.733 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.733 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:21.733 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.733 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.734 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.735 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.257261_429.1", 2020-11-19T03:06:21.735 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.735 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:21.735 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.257261094Z", 2020-11-19T03:06:21.735 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.736 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.736 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:21.736 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.736 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.737 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:21.737 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.737 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.737 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.738 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.738 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.738 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.271719_430.1", 2020-11-19T03:06:21.739 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.739 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:21.739 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.271719934Z", 2020-11-19T03:06:21.739 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.739 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.740 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:21.740 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.740 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.740 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:21.741 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.741 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.741 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.742 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.348435_431.1", 2020-11-19T03:06:21.742 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.743 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:21.743 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.348435126Z", 2020-11-19T03:06:21.743 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.743 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.744 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:21.744 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.744 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.744 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:21.745 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.745 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.745 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.746 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.746 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.358783_432.1", 2020-11-19T03:06:21.746 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.747 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:21.747 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.358783093Z", 2020-11-19T03:06:21.747 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.747 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.747 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:21.748 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.748 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.748 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:21.748 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.749 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.749 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.749 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.749 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.750 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.436658_433.1", 2020-11-19T03:06:21.750 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.750 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:21.751 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.436658111Z", 2020-11-19T03:06:21.751 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.751 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.751 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.752 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.752 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.752 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:21.752 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.753 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.753 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.753 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.753 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.754 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.443801_434.1", 2020-11-19T03:06:21.754 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.754 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:21.755 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.443801075Z", 2020-11-19T03:06:21.755 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.755 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.755 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.755 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.756 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.756 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:21.756 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.757 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.757 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.757 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.757 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.758 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.447377_435.1", 2020-11-19T03:06:21.758 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.758 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30", 2020-11-19T03:06:21.759 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.447377487Z", 2020-11-19T03:06:21.759 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.759 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.759 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.759 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.760 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.760 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_mM8-TW-E_epjEOhv1WllsAD", 2020-11-19T03:06:21.760 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.761 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.761 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.762 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.452011_436.1", 2020-11-19T03:06:21.762 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.762 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30", 2020-11-19T03:06:21.762 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.452011568Z", 2020-11-19T03:06:21.763 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.763 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.763 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.763 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.764 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.764 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_mM8-TW-E_epjEOhv1WllsAD", 2020-11-19T03:06:21.764 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.764 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.765 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.765 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.765 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.766 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.486369_437.1", 2020-11-19T03:06:21.766 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.766 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:21.766 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.486369972Z", 2020-11-19T03:06:21.767 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.767 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.767 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.767 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.768 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.768 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:21.768 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.768 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.769 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.769 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.769 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.770 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.490281_438.1", 2020-11-19T03:06:21.770 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.770 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:21.770 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.490281477Z", 2020-11-19T03:06:21.770 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.771 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.771 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.771 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.771 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.772 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:21.772 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.772 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.773 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.773 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.773 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.774 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.492125_439.1", 2020-11-19T03:06:21.774 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.774 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31", 2020-11-19T03:06:21.774 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.492125730Z", 2020-11-19T03:06:21.774 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.776 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gXleTzFvDf4A5fTuHAWH5iB", 2020-11-19T03:06:21.776 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.776 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.776 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.777 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.495843_440.1", 2020-11-19T03:06:21.778 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.778 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31", 2020-11-19T03:06:21.778 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.495843939Z", 2020-11-19T03:06:21.778 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.778 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.779 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.779 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.779 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.779 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gXleTzFvDf4A5fTuHAWH5iB", 2020-11-19T03:06:21.780 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.780 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.780 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.780 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.781 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.528063_441.1", 2020-11-19T03:06:21.781 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.782 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:21.782 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.528063552Z", 2020-11-19T03:06:21.782 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.782 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.783 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.783 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.783 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.783 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:21.784 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.784 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.784 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.785 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.532035_442.1", 2020-11-19T03:06:21.785 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.786 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:21.786 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.532035043Z", 2020-11-19T03:06:21.786 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.786 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.786 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.787 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.787 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.787 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:21.787 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.788 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.788 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.788 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.788 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.789 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.534111_443.1", 2020-11-19T03:06:21.789 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.789 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32", 2020-11-19T03:06:21.790 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.534111357Z", 2020-11-19T03:06:21.790 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.790 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.790 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.790 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.791 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.791 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_1MkPrP26FSf3qf6aGsjMwfM", 2020-11-19T03:06:21.791 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.792 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.792 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.793 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.544040_444.1", 2020-11-19T03:06:21.793 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.793 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32", 2020-11-19T03:06:21.794 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.544040853Z", 2020-11-19T03:06:21.794 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.794 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.794 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.794 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.795 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.795 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_1MkPrP26FSf3qf6aGsjMwfM", 2020-11-19T03:06:21.795 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.796 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.796 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.797 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.575866_445.1", 2020-11-19T03:06:21.797 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.797 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:21.797 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.575866003Z", 2020-11-19T03:06:21.798 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.798 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.798 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:21.798 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.799 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.799 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:21.799 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.799 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.800 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.801 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.585216_446.1", 2020-11-19T03:06:21.801 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.801 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:21.801 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.585216241Z", 2020-11-19T03:06:21.801 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.802 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.802 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:21.802 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.802 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.803 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:21.803 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.803 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.803 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.804 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.804 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.804 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.639139_447.1", 2020-11-19T03:06:21.805 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.805 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:21.805 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.639139709Z", 2020-11-19T03:06:21.805 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.806 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.806 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:21.806 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.806 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.807 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:21.807 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.807 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.807 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.808 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.808 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.808 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.649165_448.1", 2020-11-19T03:06:21.809 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.809 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:21.809 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.649165203Z", 2020-11-19T03:06:21.809 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.809 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.810 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:21.810 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.810 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.810 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:21.811 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.811 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.811 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.811 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.812 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.812 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.710088_449.1", 2020-11-19T03:06:21.812 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.813 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:21.813 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.710088485Z", 2020-11-19T03:06:21.813 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.813 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.813 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:21.814 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.814 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.814 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:21.814 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.815 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.815 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.815 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.816 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.816 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.720549_450.1", 2020-11-19T03:06:21.816 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.817 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:21.817 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.720549878Z", 2020-11-19T03:06:21.817 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.817 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.817 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:21.818 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.818 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.818 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:21.818 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.819 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.819 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.820 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.794865_451.1", 2020-11-19T03:06:21.820 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.820 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:21.821 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.794865388Z", 2020-11-19T03:06:21.821 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.821 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.821 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.822 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.822 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.822 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:21.822 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.823 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.823 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.823 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.823 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.824 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.799032_452.1", 2020-11-19T03:06:21.824 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.824 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:21.825 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.799032992Z", 2020-11-19T03:06:21.825 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.825 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.825 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.825 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.826 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.826 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:21.826 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.827 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.827 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.827 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.827 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.828 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.801245_453.1", 2020-11-19T03:06:21.828 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.828 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33", 2020-11-19T03:06:21.828 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.801245355Z", 2020-11-19T03:06:21.829 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.829 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.829 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.830 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.830 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8lKVoHs2FBMY1x8WlQGeuNM", 2020-11-19T03:06:21.830 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.831 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.831 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.832 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.805695_454.1", 2020-11-19T03:06:21.832 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.832 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33", 2020-11-19T03:06:21.832 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.805695695Z", 2020-11-19T03:06:21.833 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.833 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.833 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.833 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.834 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.834 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8lKVoHs2FBMY1x8WlQGeuNM", 2020-11-19T03:06:21.834 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.834 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.835 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.836 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.832169_455.1", 2020-11-19T03:06:21.836 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.836 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:21.836 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.832169972Z", 2020-11-19T03:06:21.836 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.837 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.837 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.837 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.837 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.838 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:21.838 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.838 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.838 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.839 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.839 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.840 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.837510_456.1", 2020-11-19T03:06:21.840 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.840 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:21.840 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.837510588Z", 2020-11-19T03:06:21.840 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.841 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.841 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.841 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.841 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.842 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:21.842 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.842 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.842 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.843 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.843 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.843 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.839242_457.1", 2020-11-19T03:06:21.844 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.844 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34", 2020-11-19T03:06:21.844 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.839242953Z", 2020-11-19T03:06:21.844 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.844 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.845 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.845 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.845 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.845 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uX_C_amZ6wHaMpUOaDbLhIh", 2020-11-19T03:06:21.846 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.846 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.846 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.846 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.847 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.847 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.843268_458.1", 2020-11-19T03:06:21.848 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.848 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34", 2020-11-19T03:06:21.848 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.843268836Z", 2020-11-19T03:06:21.848 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.848 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.849 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.849 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.849 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.849 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uX_C_amZ6wHaMpUOaDbLhIh", 2020-11-19T03:06:21.850 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.851 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.851 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.851 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.852 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.852 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.918282_459.1", 2020-11-19T03:06:21.853 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.853 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:21.853 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.918282306Z", 2020-11-19T03:06:21.853 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.853 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.854 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.854 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.854 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.854 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:21.855 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.856 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.856 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.927511_460.1", 2020-11-19T03:06:21.856 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.857 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:21.857 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.927511062Z", 2020-11-19T03:06:21.857 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.857 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.858 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.858 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.858 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.858 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:21.859 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.859 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.859 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.859 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.860 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.930242_461.1", 2020-11-19T03:06:21.860 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.861 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35", 2020-11-19T03:06:21.861 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.930242283Z", 2020-11-19T03:06:21.861 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.861 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.862 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.862 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.862 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.862 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LIlIyIKf54pQyVwNBj8qRry", 2020-11-19T03:06:21.863 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.863 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.863 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.864 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.864 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.935757_462.1", 2020-11-19T03:06:21.864 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.865 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35", 2020-11-19T03:06:21.865 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.935757929Z", 2020-11-19T03:06:21.865 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.865 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.866 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.866 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.866 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.866 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LIlIyIKf54pQyVwNBj8qRry", 2020-11-19T03:06:21.867 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.867 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.867 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.867 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.868 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.868 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.960222_463.1", 2020-11-19T03:06:21.868 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.869 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:21.869 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.960222969Z", 2020-11-19T03:06:21.869 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.869 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.874 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:21.874 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.874 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.875 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:21.875 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.875 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.875 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.876 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.876 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.876 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.966451_464.1", 2020-11-19T03:06:21.877 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.877 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:21.877 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.966451383Z", 2020-11-19T03:06:21.877 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.878 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.878 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:21.878 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.878 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.879 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:21.879 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.879 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.879 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.880 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.880 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755036.023079_465.1", 2020-11-19T03:06:21.881 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.881 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:21.881 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:56.023079326Z", 2020-11-19T03:06:21.881 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.881 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.882 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:21.882 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.882 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.883 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:21.883 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.883 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.883 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.884 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755036.036438_466.1", 2020-11-19T03:06:21.885 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.885 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:21.885 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:56.036438127Z", 2020-11-19T03:06:21.885 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.885 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.886 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:21.886 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.886 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.886 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:21.887 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.887 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.887 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.888 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.098379_467.1", 2020-11-19T03:06:21.889 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.889 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:21.889 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.098379164Z", 2020-11-19T03:06:21.889 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.889 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.890 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:21.890 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.890 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.891 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:21.891 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.891 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.892 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.893 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.106961_468.1", 2020-11-19T03:06:21.893 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.893 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:21.893 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.106961657Z", 2020-11-19T03:06:21.894 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.894 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.894 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:21.894 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.895 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.895 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:21.895 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:21.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.895 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.896 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.896 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.896 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.897 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.187937_469.1", 2020-11-19T03:06:21.897 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.897 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36", 2020-11-19T03:06:21.897 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.187937642Z", 2020-11-19T03:06:21.898 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.898 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.898 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.898 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.899 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.899 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jRizeyH8HQBihpZqQ7H0cWx", 2020-11-19T03:06:21.899 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.899 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.900 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.900 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.900 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.900 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.902 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.191921_470.1", 2020-11-19T03:06:21.902 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.902 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36", 2020-11-19T03:06:21.902 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.191921477Z", 2020-11-19T03:06:21.902 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.903 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.903 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.903 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.903 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.904 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jRizeyH8HQBihpZqQ7H0cWx", 2020-11-19T03:06:21.904 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.904 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.905 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.905 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.905 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.905 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.906 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.193772_471.1", 2020-11-19T03:06:21.906 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.906 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-36", 2020-11-19T03:06:21.906 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.193772231Z", 2020-11-19T03:06:21.906 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.907 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.907 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.907 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.907 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.908 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kYTvwOB6WrelYhgfkJRvPMC", 2020-11-19T03:06:21.908 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.908 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.908 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.909 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.909 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.909 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.910 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.197598_472.1", 2020-11-19T03:06:21.910 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.910 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-36", 2020-11-19T03:06:21.910 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.197598901Z", 2020-11-19T03:06:21.910 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.911 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.911 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.911 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.911 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.912 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kYTvwOB6WrelYhgfkJRvPMC", 2020-11-19T03:06:21.912 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.912 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.913 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.913 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.913 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.913 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.914 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.235086_473.1", 2020-11-19T03:06:21.914 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.914 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37", 2020-11-19T03:06:21.914 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.235086410Z", 2020-11-19T03:06:21.914 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.915 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.915 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.915 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.915 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.916 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_my5SPqbG7dQQSEBcU42t2-N", 2020-11-19T03:06:21.916 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.916 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.916 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.917 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.239751_474.1", 2020-11-19T03:06:21.918 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.918 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37", 2020-11-19T03:06:21.918 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.239751394Z", 2020-11-19T03:06:21.918 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.919 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.919 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.919 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.919 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.920 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_my5SPqbG7dQQSEBcU42t2-N", 2020-11-19T03:06:21.920 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.920 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.920 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.921 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.921 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.921 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.921 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.921 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.242234_475.1", 2020-11-19T03:06:21.922 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.922 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-37", 2020-11-19T03:06:21.922 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.242234716Z", 2020-11-19T03:06:21.922 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.923 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.923 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.923 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.923 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.924 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_g91x9CrlSKmRRavm4MwY2r5", 2020-11-19T03:06:21.924 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.924 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.924 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.925 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.925 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.925 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.247176_476.1", 2020-11-19T03:06:21.926 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.926 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-37", 2020-11-19T03:06:21.926 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.247176520Z", 2020-11-19T03:06:21.926 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.926 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.927 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.927 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.927 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.927 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_g91x9CrlSKmRRavm4MwY2r5", 2020-11-19T03:06:21.928 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.928 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.928 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.928 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.929 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.929 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755038.595496_477.1", 2020-11-19T03:06:21.929 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.930 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38", 2020-11-19T03:06:21.930 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.595496690Z", 2020-11-19T03:06:21.930 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.930 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.930 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V7n32juggP5QmeLeOGxy3UK", 2020-11-19T03:06:21.931 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.933 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755038.599524_478.1", 2020-11-19T03:06:21.933 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.934 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38", 2020-11-19T03:06:21.934 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.599524203Z", 2020-11-19T03:06:21.934 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.934 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.934 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.935 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.935 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.935 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V7n32juggP5QmeLeOGxy3UK", 2020-11-19T03:06:21.935 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.936 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.936 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.937 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755038.601388_479.1", 2020-11-19T03:06:21.937 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.937 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-38", 2020-11-19T03:06:21.938 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.601388404Z", 2020-11-19T03:06:21.938 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.938 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.938 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.938 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.939 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.939 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_W1ZCfu17NUxLx7oS231exeH", 2020-11-19T03:06:21.939 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.939 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.940 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.940 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.940 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.941 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755038.605369_480.1", 2020-11-19T03:06:21.941 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.941 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-38", 2020-11-19T03:06:21.941 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.605369575Z", 2020-11-19T03:06:21.941 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.942 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.942 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.942 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.942 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.943 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_W1ZCfu17NUxLx7oS231exeH", 2020-11-19T03:06:21.944 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.945 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.945 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.946 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.946 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.947 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.318171_485.1", 2020-11-19T03:06:21.948 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.948 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-39:d71482d6-15fd-4715-9992-a25958dcb538.4341.39", 2020-11-19T03:06:21.949 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.318171704Z", 2020-11-19T03:06:21.949 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.949 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.950 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.950 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.951 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.952 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MR6DDMoW-bWzeQ2lVYjkv_T", 2020-11-19T03:06:21.952 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.953 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.953 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.954 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.954 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.955 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.326000_486.1", 2020-11-19T03:06:21.956 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.956 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-39:d71482d6-15fd-4715-9992-a25958dcb538.4341.39", 2020-11-19T03:06:21.957 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.326000603Z", 2020-11-19T03:06:21.957 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.957 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.958 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.958 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.959 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.959 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MR6DDMoW-bWzeQ2lVYjkv_T", 2020-11-19T03:06:21.959 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.960 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.960 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.961 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.961 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.962 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.328429_487.1", 2020-11-19T03:06:21.962 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.963 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-39", 2020-11-19T03:06:21.964 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.328429735Z", 2020-11-19T03:06:21.964 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.965 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.965 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.966 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.966 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.967 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2xs16O_Y3iMqtaOm0GzhjDK", 2020-11-19T03:06:21.967 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.968 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.968 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.970 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.333402_488.1", 2020-11-19T03:06:21.970 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.971 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-39", 2020-11-19T03:06:21.971 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.333402182Z", 2020-11-19T03:06:21.971 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.971 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.972 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.972 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.974 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.974 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2xs16O_Y3iMqtaOm0GzhjDK", 2020-11-19T03:06:21.975 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.975 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.976 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.978 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.366052_489.1", 2020-11-19T03:06:21.979 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.979 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-40:d71482d6-15fd-4715-9992-a25958dcb538.4341.40", 2020-11-19T03:06:21.980 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.366052352Z", 2020-11-19T03:06:21.980 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.981 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.981 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.982 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.983 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.983 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_oGE_O83RX1iSJnFe2p4H0J-", 2020-11-19T03:06:21.983 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.984 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.985 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:21.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.986 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.372774_490.1", 2020-11-19T03:06:21.987 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:21.987 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-40:d71482d6-15fd-4715-9992-a25958dcb538.4341.40", 2020-11-19T03:06:21.987 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.372774679Z", 2020-11-19T03:06:21.988 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.989 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.990 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.991 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.992 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.992 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_oGE_O83RX1iSJnFe2p4H0J-", 2020-11-19T03:06:21.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:21.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.993 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:21.994 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:21.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:21.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:21.996 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.375737_491.1", 2020-11-19T03:06:21.996 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:21.996 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-40", 2020-11-19T03:06:21.997 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.375737830Z", 2020-11-19T03:06:21.997 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:21.997 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:21.998 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:21.998 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:21.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:21.999 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:21.999 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bGn8OsdsLasWP-8EeCEuz7n", 2020-11-19T03:06:21.999 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.000 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.000 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.001 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.001 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.002 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.382371_492.1", 2020-11-19T03:06:22.002 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.002 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-40", 2020-11-19T03:06:22.003 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.382371633Z", 2020-11-19T03:06:22.003 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.003 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.004 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.004 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.005 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.005 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bGn8OsdsLasWP-8EeCEuz7n", 2020-11-19T03:06:22.005 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.006 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.006 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.008 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.429548_493.1", 2020-11-19T03:06:22.008 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.008 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-41:d71482d6-15fd-4715-9992-a25958dcb538.4341.41", 2020-11-19T03:06:22.009 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.429548543Z", 2020-11-19T03:06:22.009 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.009 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.010 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.010 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.010 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.011 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__wSJMtQIou6xYhPPtSS8YuA", 2020-11-19T03:06:22.011 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.011 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.011 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.012 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.436987_494.1", 2020-11-19T03:06:22.013 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.013 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-41:d71482d6-15fd-4715-9992-a25958dcb538.4341.41", 2020-11-19T03:06:22.013 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.436987958Z", 2020-11-19T03:06:22.014 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.014 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.014 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.014 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.015 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.015 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__wSJMtQIou6xYhPPtSS8YuA", 2020-11-19T03:06:22.016 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.017 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.017 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.017 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.017 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.018 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.439804_495.1", 2020-11-19T03:06:22.019 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.019 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-41", 2020-11-19T03:06:22.019 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.439804366Z", 2020-11-19T03:06:22.020 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.020 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.020 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.020 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.021 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.021 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_RsdLOBWehIFSw4jrAQphm1V", 2020-11-19T03:06:22.021 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.023 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.023 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.023 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.024 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.025 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.443579_496.1", 2020-11-19T03:06:22.025 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.025 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-41", 2020-11-19T03:06:22.026 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.443579496Z", 2020-11-19T03:06:22.026 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.026 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.027 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.027 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.027 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.028 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_RsdLOBWehIFSw4jrAQphm1V", 2020-11-19T03:06:22.028 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.029 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.029 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.030 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.031 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.049424_71.1", 2020-11-19T03:06:22.031 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.031 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-1:d71482d6-15fd-4715-9992-a25958dcb538.4341.1", 2020-11-19T03:06:22.032 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.049424391Z", 2020-11-19T03:06:22.032 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.032 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.032 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.033 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.033 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.034 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qgItci47XzVPPEUbRm9FP71", 2020-11-19T03:06:22.034 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.035 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.035 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.036 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.036 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.037 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.067007_75.1", 2020-11-19T03:06:22.037 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.037 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-1:d71482d6-15fd-4715-9992-a25958dcb538.4341.1", 2020-11-19T03:06:22.038 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.067007384Z", 2020-11-19T03:06:22.038 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.038 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.039 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.039 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.040 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.040 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qgItci47XzVPPEUbRm9FP71", 2020-11-19T03:06:22.040 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.041 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.041 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.042 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.042 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.043 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.068631_76.1", 2020-11-19T03:06:22.043 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.043 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-1", 2020-11-19T03:06:22.044 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.068631964Z", 2020-11-19T03:06:22.044 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.044 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.044 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.045 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.046 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.046 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_94mQ_WxhbwDk-kFSjKqwmYF", 2020-11-19T03:06:22.046 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.047 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.047 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.053 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.054 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.056 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.071728_77.1", 2020-11-19T03:06:22.056 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.057 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-1", 2020-11-19T03:06:22.057 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.071728952Z", 2020-11-19T03:06:22.057 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.058 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.058 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.061 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.062 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.062 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_94mQ_WxhbwDk-kFSjKqwmYF", 2020-11-19T03:06:22.062 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.063 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.063 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.063 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.063 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.064 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.094013_78.1", 2020-11-19T03:06:22.065 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.065 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-2:d71482d6-15fd-4715-9992-a25958dcb538.4341.2", 2020-11-19T03:06:22.065 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.094013583Z", 2020-11-19T03:06:22.065 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.066 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.067 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.067 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.068 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bSYBD3JTM-OkT97a99BJ1cL", 2020-11-19T03:06:22.069 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.069 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.070 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.071 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.098511_79.1", 2020-11-19T03:06:22.071 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.071 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-2:d71482d6-15fd-4715-9992-a25958dcb538.4341.2", 2020-11-19T03:06:22.072 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.098511431Z", 2020-11-19T03:06:22.072 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.072 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.072 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.073 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.073 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.073 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bSYBD3JTM-OkT97a99BJ1cL", 2020-11-19T03:06:22.074 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.074 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.074 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.075 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.100851_80.1", 2020-11-19T03:06:22.076 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.076 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-2", 2020-11-19T03:06:22.076 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.100851568Z", 2020-11-19T03:06:22.076 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.076 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.077 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.077 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.077 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.077 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5UuoR0SlRK-wajXJjy8RaHm", 2020-11-19T03:06:22.078 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.078 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.079 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.079 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.080 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.105586_81.1", 2020-11-19T03:06:22.080 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.081 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-2", 2020-11-19T03:06:22.081 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.105586168Z", 2020-11-19T03:06:22.081 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.081 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.082 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.082 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.082 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.082 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5UuoR0SlRK-wajXJjy8RaHm", 2020-11-19T03:06:22.083 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.083 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.083 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.084 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.084 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.085 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754538.238498_82.1", 2020-11-19T03:06:22.085 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.085 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-3:d71482d6-15fd-4715-9992-a25958dcb538.4341.3", 2020-11-19T03:06:22.085 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.238498971Z", 2020-11-19T03:06:22.086 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.086 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.086 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.086 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.087 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.087 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_GAW34u3sdJwjuChdK6wHKmm", 2020-11-19T03:06:22.087 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.087 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.088 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.088 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.088 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.089 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754538.263102_83.1", 2020-11-19T03:06:22.089 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.089 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-3:d71482d6-15fd-4715-9992-a25958dcb538.4341.3", 2020-11-19T03:06:22.089 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.263102074Z", 2020-11-19T03:06:22.090 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.090 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.090 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.090 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.091 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.091 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_GAW34u3sdJwjuChdK6wHKmm", 2020-11-19T03:06:22.091 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.092 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.092 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.092 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.093 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754538.265151_84.1", 2020-11-19T03:06:22.093 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.094 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-3", 2020-11-19T03:06:22.094 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.265151277Z", 2020-11-19T03:06:22.094 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.094 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.094 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.095 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.095 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.095 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tL7nc-Jhsvjv2EEZRXT73OW", 2020-11-19T03:06:22.095 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.096 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.096 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.096 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.096 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.097 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754538.270230_85.1", 2020-11-19T03:06:22.097 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.098 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-3", 2020-11-19T03:06:22.098 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.270230968Z", 2020-11-19T03:06:22.098 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.098 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.099 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.099 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.100 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.100 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tL7nc-Jhsvjv2EEZRXT73OW", 2020-11-19T03:06:22.100 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.101 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.101 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.102 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.102 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.646238_86.1", 2020-11-19T03:06:22.103 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.103 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-4:d71482d6-15fd-4715-9992-a25958dcb538.4341.4", 2020-11-19T03:06:22.103 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.646238073Z", 2020-11-19T03:06:22.104 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.104 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.105 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.105 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.106 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kXyu9LGpqsO6w8QSZAhud3g", 2020-11-19T03:06:22.106 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.107 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.107 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.107 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.108 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.109 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.653311_88.1", 2020-11-19T03:06:22.109 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.110 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-4:d71482d6-15fd-4715-9992-a25958dcb538.4341.4", 2020-11-19T03:06:22.110 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.653311568Z", 2020-11-19T03:06:22.110 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.110 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.111 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.111 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.111 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.112 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kXyu9LGpqsO6w8QSZAhud3g", 2020-11-19T03:06:22.112 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.112 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.112 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.113 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.113 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.113 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.655601_89.1", 2020-11-19T03:06:22.114 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.114 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-4", 2020-11-19T03:06:22.114 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.655601491Z", 2020-11-19T03:06:22.114 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.114 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.115 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.115 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.115 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.116 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tMmjLsHDWqdVdqcqT0YXzke", 2020-11-19T03:06:22.116 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.116 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.116 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.117 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.117 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.118 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.661306_90.1", 2020-11-19T03:06:22.118 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.118 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-4", 2020-11-19T03:06:22.118 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.661306950Z", 2020-11-19T03:06:22.119 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.119 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.119 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.119 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.120 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.120 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tMmjLsHDWqdVdqcqT0YXzke", 2020-11-19T03:06:22.120 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.120 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.121 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.121 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.121 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.122 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.685988_91.1", 2020-11-19T03:06:22.122 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.123 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-5:d71482d6-15fd-4715-9992-a25958dcb538.4341.5", 2020-11-19T03:06:22.123 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.685988313Z", 2020-11-19T03:06:22.123 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.123 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.124 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qVWvmOqSOj4nJ_IyDQjqQEP", 2020-11-19T03:06:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.126 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.126 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.126 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.127 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.128 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.690669_92.1", 2020-11-19T03:06:22.128 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.128 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-5:d71482d6-15fd-4715-9992-a25958dcb538.4341.5", 2020-11-19T03:06:22.128 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.690669131Z", 2020-11-19T03:06:22.128 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.129 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.129 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.129 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.130 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.130 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qVWvmOqSOj4nJ_IyDQjqQEP", 2020-11-19T03:06:22.130 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.131 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.131 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.131 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.131 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.132 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.693071_93.1", 2020-11-19T03:06:22.132 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.133 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-5", 2020-11-19T03:06:22.133 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.693071890Z", 2020-11-19T03:06:22.133 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.133 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.133 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.134 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.134 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.134 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IRRwWfE5ELTX6B9PeD0Wss2", 2020-11-19T03:06:22.135 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.135 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.135 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.135 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.136 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.136 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.697214_94.1", 2020-11-19T03:06:22.137 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.137 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-5", 2020-11-19T03:06:22.137 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.697214643Z", 2020-11-19T03:06:22.137 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.138 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.139 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.139 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.139 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.140 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IRRwWfE5ELTX6B9PeD0Wss2", 2020-11-19T03:06:22.140 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.140 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.141 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.141 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.141 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.142 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.740644_95.1", 2020-11-19T03:06:22.142 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.142 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-6:d71482d6-15fd-4715-9992-a25958dcb538.4341.6", 2020-11-19T03:06:22.142 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.740644196Z", 2020-11-19T03:06:22.143 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.143 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.143 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.143 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.144 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.144 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qrn-62ONHSx8esDlsD49N2S", 2020-11-19T03:06:22.144 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.145 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.145 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.145 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.146 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.146 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.744532_96.1", 2020-11-19T03:06:22.146 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.147 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-6:d71482d6-15fd-4715-9992-a25958dcb538.4341.6", 2020-11-19T03:06:22.147 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.744532390Z", 2020-11-19T03:06:22.147 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.147 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.148 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.148 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.148 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.148 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qrn-62ONHSx8esDlsD49N2S", 2020-11-19T03:06:22.149 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.149 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.149 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.150 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.150 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.746440_97.1", 2020-11-19T03:06:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-6", 2020-11-19T03:06:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.746440476Z", 2020-11-19T03:06:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_wFarfv5IWniL3C-MOrXA7X0", 2020-11-19T03:06:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.750854_98.1", 2020-11-19T03:06:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-6", 2020-11-19T03:06:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.750854665Z", 2020-11-19T03:06:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_wFarfv5IWniL3C-MOrXA7X0", 2020-11-19T03:06:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.678670_99.1", 2020-11-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7:d71482d6-15fd-4715-9992-a25958dcb538.4341.7", 2020-11-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.678670320Z", 2020-11-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3L9q5SfSgUsZej9VYTkemdm", 2020-11-19T03:06:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.684139_102.1", 2020-11-19T03:06:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7:d71482d6-15fd-4715-9992-a25958dcb538.4341.7", 2020-11-19T03:06:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.684139704Z", 2020-11-19T03:06:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3L9q5SfSgUsZej9VYTkemdm", 2020-11-19T03:06:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.686054_103.1", 2020-11-19T03:06:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7", 2020-11-19T03:06:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.686054368Z", 2020-11-19T03:06:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.690172_104.1", 2020-11-19T03:06:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7", 2020-11-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.690172550Z", 2020-11-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.714743_105.1", 2020-11-19T03:06:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8:d71482d6-15fd-4715-9992-a25958dcb538.4341.8", 2020-11-19T03:06:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.714743025Z", 2020-11-19T03:06:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Y2tOicXm39NSETUiNVpNZLc", 2020-11-19T03:06:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.719653_106.1", 2020-11-19T03:06:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8:d71482d6-15fd-4715-9992-a25958dcb538.4341.8", 2020-11-19T03:06:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.719653546Z", 2020-11-19T03:06:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Y2tOicXm39NSETUiNVpNZLc", 2020-11-19T03:06:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.722344_107.1", 2020-11-19T03:06:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8", 2020-11-19T03:06:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.722344069Z", 2020-11-19T03:06:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.727853_108.1", 2020-11-19T03:06:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8", 2020-11-19T03:06:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.727853163Z", 2020-11-19T03:06:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.773887_109.1", 2020-11-19T03:06:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9:d71482d6-15fd-4715-9992-a25958dcb538.4341.9", 2020-11-19T03:06:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.773887989Z", 2020-11-19T03:06:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vGJjA-wwP6rD2bH8RFgbNPS", 2020-11-19T03:06:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.778690_110.1", 2020-11-19T03:06:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9:d71482d6-15fd-4715-9992-a25958dcb538.4341.9", 2020-11-19T03:06:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.778690126Z", 2020-11-19T03:06:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vGJjA-wwP6rD2bH8RFgbNPS", 2020-11-19T03:06:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.781507_111.1", 2020-11-19T03:06:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9", 2020-11-19T03:06:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.781507411Z", 2020-11-19T03:06:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.787262_112.1", 2020-11-19T03:06:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9", 2020-11-19T03:06:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.787262520Z", 2020-11-19T03:06:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.203 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.203 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:22.203 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:22.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.203 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.204 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:22.204 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.204 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.204 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.205 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.124200_113.1", 2020-11-19T03:06:22.206 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.206 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7", 2020-11-19T03:06:22.206 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.124200729Z", 2020-11-19T03:06:22.206 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.206 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.206 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:22.207 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.207 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.207 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:22.207 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:22.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.208 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.208 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2020-11-19T03:06:22.208 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.208 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.209 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.128855_115.1", 2020-11-19T03:06:22.209 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.209 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7", 2020-11-19T03:06:22.210 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.128855513Z", 2020-11-19T03:06:22.210 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.210 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.210 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:22.210 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.211 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.211 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:22.211 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:22.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.212 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.212 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.213 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.183608_116.1", 2020-11-19T03:06:22.213 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.213 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8", 2020-11-19T03:06:22.213 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.183608180Z", 2020-11-19T03:06:22.214 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.214 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.214 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:22.214 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.215 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.215 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:22.215 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:22.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.215 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.216 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2020-11-19T03:06:22.216 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.216 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.216 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.217 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.189052_117.1", 2020-11-19T03:06:22.217 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.217 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8", 2020-11-19T03:06:22.217 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.189052423Z", 2020-11-19T03:06:22.217 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.218 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.218 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:22.219 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.219 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.220 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.220 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:22.220 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:22.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.220 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.221 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.221 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.221 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.221 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.221 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.222 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.288556_118.1", 2020-11-19T03:06:22.222 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.222 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9", 2020-11-19T03:06:22.222 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.288556017Z", 2020-11-19T03:06:22.222 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.223 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.223 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:22.223 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.223 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.224 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:22.224 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:22.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.224 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.224 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2020-11-19T03:06:22.225 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.225 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.225 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.225 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:22.225 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.294580_119.1", 2020-11-19T03:06:22.226 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:22.226 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9", 2020-11-19T03:06:22.226 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580366Z", 2020-11-19T03:06:22.226 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:22.226 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:22.227 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:22.227 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:22.227 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.227 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:22.227 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:22.228 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:22.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:22.228 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:22.228 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:22.228 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.228 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.229 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:22.229 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:06:22.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:22.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:22.232 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b2c8afed-318c-484c-8dd4-daab7a757391 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:22.272 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:22.519 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:22.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.519 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754414.632290_106.1", 2020-11-19T03:06:22.520 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "user", 2020-11-19T03:06:22.520 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "rgw-multisite-test-user", 2020-11-19T03:06:22.520 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.632290363Z", 2020-11-19T03:06:22.520 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.520 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.521 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.521 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.521 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.522 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_OD787OEqCb37r-dWrIu2W6g", 2020-11-19T03:06:22.522 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.522 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.522 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.523 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.523 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.523 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754414.640760_109.1", 2020-11-19T03:06:22.524 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "user", 2020-11-19T03:06:22.524 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "rgw-multisite-test-user", 2020-11-19T03:06:22.524 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:53:34.640760721Z", 2020-11-19T03:06:22.524 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.525 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.525 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.525 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.525 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.526 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_OD787OEqCb37r-dWrIu2W6g", 2020-11-19T03:06:22.526 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.526 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.527 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.528 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.755439_114.1", 2020-11-19T03:06:22.528 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.528 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10", 2020-11-19T03:06:22.528 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.755439437Z", 2020-11-19T03:06:22.528 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.529 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.529 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.529 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.529 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.532 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jDQMSP1imZk9M53g_LemGsD", 2020-11-19T03:06:22.532 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.532 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.533 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.534 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.759756_127.1", 2020-11-19T03:06:22.534 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.534 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10", 2020-11-19T03:06:22.535 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.759756660Z", 2020-11-19T03:06:22.535 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.535 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.535 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.535 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.539 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.539 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jDQMSP1imZk9M53g_LemGsD", 2020-11-19T03:06:22.539 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.540 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.540 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.541 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.761697_128.1", 2020-11-19T03:06:22.541 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.541 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-10", 2020-11-19T03:06:22.542 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.761697955Z", 2020-11-19T03:06:22.542 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.542 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.542 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.542 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.543 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.543 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_QaR_8xY7tB3j-5_fwfgIAp2", 2020-11-19T03:06:22.543 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.544 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.544 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.545 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.765274_129.1", 2020-11-19T03:06:22.545 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.545 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-10", 2020-11-19T03:06:22.545 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.765274847Z", 2020-11-19T03:06:22.549 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.550 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.550 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.550 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.551 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.551 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_QaR_8xY7tB3j-5_fwfgIAp2", 2020-11-19T03:06:22.551 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.552 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.552 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.552 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.552 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.553 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.789431_130.1", 2020-11-19T03:06:22.553 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.553 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11", 2020-11-19T03:06:22.553 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.789431860Z", 2020-11-19T03:06:22.554 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.554 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.554 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.554 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.555 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.555 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_h4zkJ0TD5OOIK8n5LNF_3YA", 2020-11-19T03:06:22.555 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.555 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.556 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.557 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.793774_131.1", 2020-11-19T03:06:22.557 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.558 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11", 2020-11-19T03:06:22.558 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.793774959Z", 2020-11-19T03:06:22.558 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.558 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.559 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.559 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.559 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.559 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_h4zkJ0TD5OOIK8n5LNF_3YA", 2020-11-19T03:06:22.560 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.560 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.560 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.561 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.561 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.798353_132.1", 2020-11-19T03:06:22.562 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.562 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-11", 2020-11-19T03:06:22.562 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.798353113Z", 2020-11-19T03:06:22.563 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.563 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.563 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.563 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.564 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.564 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5lbvhy-2ssHalMo0XAyb9jK", 2020-11-19T03:06:22.564 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.564 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.565 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.566 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.803735_133.1", 2020-11-19T03:06:22.566 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.566 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-11", 2020-11-19T03:06:22.566 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.803735217Z", 2020-11-19T03:06:22.566 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.567 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.569 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.570 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.571 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.576 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5lbvhy-2ssHalMo0XAyb9jK", 2020-11-19T03:06:22.581 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.592 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.593 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.593 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.594 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.597 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.854454_134.1", 2020-11-19T03:06:22.600 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.600 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12", 2020-11-19T03:06:22.600 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.854454685Z", 2020-11-19T03:06:22.600 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.601 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.601 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.601 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.602 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.602 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_DG8ZKomIxnXa3vMXcvP8ajO", 2020-11-19T03:06:22.602 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.603 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.603 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.603 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.603 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.604 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.858973_135.1", 2020-11-19T03:06:22.604 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.604 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12", 2020-11-19T03:06:22.605 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.858973052Z", 2020-11-19T03:06:22.605 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.605 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.605 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.605 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.606 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.606 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_DG8ZKomIxnXa3vMXcvP8ajO", 2020-11-19T03:06:22.606 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.607 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.607 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.608 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.608 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.609 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.860848_136.1", 2020-11-19T03:06:22.609 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.609 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-12", 2020-11-19T03:06:22.610 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.860848274Z", 2020-11-19T03:06:22.610 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.610 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.610 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.610 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.611 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.611 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_xHh-uu7OMNHE3E1tkdnCoN0", 2020-11-19T03:06:22.611 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.612 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.612 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.612 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.614 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754545.866167_137.1", 2020-11-19T03:06:22.614 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.614 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-12", 2020-11-19T03:06:22.614 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:45.866167085Z", 2020-11-19T03:06:22.615 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.615 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.615 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.615 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.616 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.616 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_xHh-uu7OMNHE3E1tkdnCoN0", 2020-11-19T03:06:22.616 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.616 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.617 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.617 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.617 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.618 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.787642_145.1", 2020-11-19T03:06:22.618 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.618 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13", 2020-11-19T03:06:22.619 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.787642625Z", 2020-11-19T03:06:22.619 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.619 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.619 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.619 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.620 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.620 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qxTzHkbF9wemeZVynRNqeCj", 2020-11-19T03:06:22.620 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.621 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.621 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.621 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.621 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.622 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.792198_150.1", 2020-11-19T03:06:22.622 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.623 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13", 2020-11-19T03:06:22.623 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.792198045Z", 2020-11-19T03:06:22.623 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.623 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.623 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.624 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.624 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.624 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qxTzHkbF9wemeZVynRNqeCj", 2020-11-19T03:06:22.624 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.625 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.625 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.626 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.626 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.626 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.794166_151.1", 2020-11-19T03:06:22.627 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.627 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-13", 2020-11-19T03:06:22.627 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.794166672Z", 2020-11-19T03:06:22.627 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.628 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.628 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.628 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.628 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.629 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ogTrNysaHhYexwepXbm3dwI", 2020-11-19T03:06:22.629 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.629 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.630 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.630 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.630 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.631 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.798068_152.1", 2020-11-19T03:06:22.631 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.631 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-13", 2020-11-19T03:06:22.631 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.798068026Z", 2020-11-19T03:06:22.632 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.632 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.632 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.632 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.633 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.633 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ogTrNysaHhYexwepXbm3dwI", 2020-11-19T03:06:22.633 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.634 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.634 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.634 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.635 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.635 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.819538_154.1", 2020-11-19T03:06:22.635 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.636 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14", 2020-11-19T03:06:22.636 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.819538205Z", 2020-11-19T03:06:22.636 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.636 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.636 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.637 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.637 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.637 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dcIW3-alzbWV6sgq13MRCG7", 2020-11-19T03:06:22.637 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.638 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.638 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.638 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.639 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.639 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.825038_155.1", 2020-11-19T03:06:22.639 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.640 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14", 2020-11-19T03:06:22.640 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.825038088Z", 2020-11-19T03:06:22.640 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.640 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.640 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.641 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.641 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.641 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dcIW3-alzbWV6sgq13MRCG7", 2020-11-19T03:06:22.641 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.642 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.642 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.642 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.642 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.643 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.827341_156.1", 2020-11-19T03:06:22.643 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.643 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-14", 2020-11-19T03:06:22.644 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.827341897Z", 2020-11-19T03:06:22.644 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.644 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.644 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.644 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.645 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.645 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_TbVBTXwBuNJEI3zmckRRIjz", 2020-11-19T03:06:22.645 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.646 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.646 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.646 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.646 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.647 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.832699_157.1", 2020-11-19T03:06:22.647 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.647 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-14", 2020-11-19T03:06:22.647 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.832699383Z", 2020-11-19T03:06:22.648 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.648 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.648 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.648 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.649 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.649 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_TbVBTXwBuNJEI3zmckRRIjz", 2020-11-19T03:06:22.649 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.649 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.650 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.651 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.868186_158.1", 2020-11-19T03:06:22.651 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.651 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15", 2020-11-19T03:06:22.651 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.868186115Z", 2020-11-19T03:06:22.652 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.652 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.652 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.652 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.653 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.653 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Pzb_ZFMDCfzlSj1lsKyhj_z", 2020-11-19T03:06:22.653 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.653 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.654 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.654 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.654 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.655 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.872546_159.1", 2020-11-19T03:06:22.655 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.655 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15", 2020-11-19T03:06:22.655 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.872546446Z", 2020-11-19T03:06:22.655 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.656 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.656 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.656 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.656 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.657 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Pzb_ZFMDCfzlSj1lsKyhj_z", 2020-11-19T03:06:22.657 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.657 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.657 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.658 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.658 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.658 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.876242_160.1", 2020-11-19T03:06:22.659 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.659 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-15", 2020-11-19T03:06:22.659 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.876242156Z", 2020-11-19T03:06:22.659 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.659 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.660 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.660 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.660 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.660 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Hkg8dq-5IxwuJ-1F2ioM3Ha", 2020-11-19T03:06:22.661 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.661 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.661 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.662 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754557.888825_161.1", 2020-11-19T03:06:22.662 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.663 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-15", 2020-11-19T03:06:22.663 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:57.888825992Z", 2020-11-19T03:06:22.663 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.663 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.663 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.664 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.664 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.664 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Hkg8dq-5IxwuJ-1F2ioM3Ha", 2020-11-19T03:06:22.664 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.665 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.665 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.665 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.665 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.666 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.798543_207.1", 2020-11-19T03:06:22.666 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.666 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:22.667 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.798543679Z", 2020-11-19T03:06:22.667 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.667 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.667 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.667 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.668 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.668 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:22.668 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.669 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.669 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.670 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.803204_211.1", 2020-11-19T03:06:22.670 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.670 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:22.670 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.803204912Z", 2020-11-19T03:06:22.671 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.671 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.671 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.671 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.672 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.672 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:22.672 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.672 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.673 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.674 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.805294_212.1", 2020-11-19T03:06:22.674 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.674 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16", 2020-11-19T03:06:22.674 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.805294677Z", 2020-11-19T03:06:22.674 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.675 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.675 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.675 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.675 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.676 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_8SdF_vsimBqHGDaqiMgkarR", 2020-11-19T03:06:22.676 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.676 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.676 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.678 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.810747_213.1", 2020-11-19T03:06:22.678 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.678 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16", 2020-11-19T03:06:22.678 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.810747502Z", 2020-11-19T03:06:22.678 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.679 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.679 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.679 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.679 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.680 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_8SdF_vsimBqHGDaqiMgkarR", 2020-11-19T03:06:22.680 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.680 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.680 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.680 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.681 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.681 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.838151_214.1", 2020-11-19T03:06:22.681 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.682 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:22.682 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.838151294Z", 2020-11-19T03:06:22.682 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.682 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.682 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.683 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.683 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.683 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:22.683 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.684 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.684 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.684 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.685 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.685 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.842707_215.1", 2020-11-19T03:06:22.685 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.686 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:22.686 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.842707380Z", 2020-11-19T03:06:22.686 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.686 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.686 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.687 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.687 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.687 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:22.687 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.688 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.688 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.688 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.688 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.689 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.844853_216.1", 2020-11-19T03:06:22.689 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.689 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17", 2020-11-19T03:06:22.690 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.844853013Z", 2020-11-19T03:06:22.690 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.690 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.690 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.690 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.691 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.691 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__nKT7kdBFED-s1dk1lgSXVH", 2020-11-19T03:06:22.691 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.692 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.692 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.693 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.849738_217.1", 2020-11-19T03:06:22.693 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.693 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17", 2020-11-19T03:06:22.693 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.849738526Z", 2020-11-19T03:06:22.694 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.694 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.694 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.694 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.695 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.695 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__nKT7kdBFED-s1dk1lgSXVH", 2020-11-19T03:06:22.695 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.696 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.696 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.697 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.905621_218.1", 2020-11-19T03:06:22.697 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.697 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:22.697 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.905621496Z", 2020-11-19T03:06:22.698 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.698 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.698 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.698 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.699 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.699 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:22.699 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.699 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.700 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:22.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.701 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.911693_219.1", 2020-11-19T03:06:22.701 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:22.701 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:22.701 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.911693551Z", 2020-11-19T03:06:22.701 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:22.702 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:22.702 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:22.702 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:22.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.702 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:22.703 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:22.703 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:22.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.703 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:22.703 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:22.704 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.704 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:22.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:22.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:22.704 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.915252_220.1", 2020-11-19T03:06:22.705 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:22.705 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18", 2020-11-19T03:06:23.393 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.915252896Z", 2020-11-19T03:06:23.394 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.394 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.394 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.395 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.395 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.395 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6l8yn5mDyDSfq7DcU_4ezuR", 2020-11-19T03:06:23.395 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.396 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.396 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.396 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.397 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.397 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.919671_221.1", 2020-11-19T03:06:23.397 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.398 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18", 2020-11-19T03:06:23.398 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.919671168Z", 2020-11-19T03:06:23.398 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.398 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.398 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.399 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.399 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.399 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6l8yn5mDyDSfq7DcU_4ezuR", 2020-11-19T03:06:23.400 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.400 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.400 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.400 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.401 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.401 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.939873_222.1", 2020-11-19T03:06:23.401 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.402 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:23.402 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.939873846Z", 2020-11-19T03:06:23.402 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.402 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.402 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:23.403 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.403 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.403 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:23.403 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.404 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.404 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.405 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.948027_223.1", 2020-11-19T03:06:23.405 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.406 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:23.406 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.948027995Z", 2020-11-19T03:06:23.406 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.406 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.406 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:23.407 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.407 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.407 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:23.407 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.408 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.408 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.409 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.955102_224.1", 2020-11-19T03:06:23.409 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.410 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:23.410 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.955102556Z", 2020-11-19T03:06:23.410 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.410 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.411 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:23.411 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.412 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.412 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:23.412 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.413 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.413 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.413 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.413 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.414 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.962377_225.1", 2020-11-19T03:06:23.414 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.415 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:23.415 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.962377954Z", 2020-11-19T03:06:23.415 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.415 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.415 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:23.416 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.416 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.416 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:23.416 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.417 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.417 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.417 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.417 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.418 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.974309_226.1", 2020-11-19T03:06:23.418 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.418 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:23.419 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.974309142Z", 2020-11-19T03:06:23.419 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.419 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.419 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:23.419 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.420 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.420 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:23.420 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.421 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.421 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.421 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.421 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.422 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754607.983290_227.1", 2020-11-19T03:06:23.422 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.422 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:23.423 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:56:47.983290859Z", 2020-11-19T03:06:23.423 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.423 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.424 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:23.424 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.424 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.424 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:23.425 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.425 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.425 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.425 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.426 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.426 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.853615_253.1", 2020-11-19T03:06:23.426 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.427 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:23.427 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.853615363Z", 2020-11-19T03:06:23.427 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.427 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.427 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.428 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.428 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.428 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:23.428 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.429 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.429 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.429 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.429 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.430 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.857577_262.1", 2020-11-19T03:06:23.430 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.430 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:23.431 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.857577608Z", 2020-11-19T03:06:23.431 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.431 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.431 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.431 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.432 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.432 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:23.432 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.433 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.433 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.433 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.433 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.434 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.859745_263.1", 2020-11-19T03:06:23.434 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.434 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19", 2020-11-19T03:06:23.434 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.859745204Z", 2020-11-19T03:06:23.435 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.435 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.435 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.435 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.436 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.436 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Qy4UpA_MiVj8fLm1yoUPwmx", 2020-11-19T03:06:23.436 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.436 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.437 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.437 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.437 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.438 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.863407_264.1", 2020-11-19T03:06:23.438 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.438 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19", 2020-11-19T03:06:23.438 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.863407887Z", 2020-11-19T03:06:23.439 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.439 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.439 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.439 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.440 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.440 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Qy4UpA_MiVj8fLm1yoUPwmx", 2020-11-19T03:06:23.440 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.440 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.441 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.441 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.441 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.442 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.869018_265.1", 2020-11-19T03:06:23.442 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.442 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:23.442 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.869018103Z", 2020-11-19T03:06:23.442 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.443 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.443 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:23.443 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.443 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.444 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:23.444 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.444 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.444 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.446 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754783.874924_266.1", 2020-11-19T03:06:23.446 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.446 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:23.446 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:43.874924462Z", 2020-11-19T03:06:23.446 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.447 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.447 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:23.447 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.447 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.448 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:23.448 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.448 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.448 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.449 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754893.333355_282.1", 2020-11-19T03:06:23.450 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.450 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:23.450 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:33.333355116Z", 2020-11-19T03:06:23.450 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.450 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.451 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.451 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.451 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.451 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.452 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.452 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.452 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.452 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.453 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754893.338774_293.1", 2020-11-19T03:06:23.453 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.454 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:23.454 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:33.338774111Z", 2020-11-19T03:06:23.454 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.454 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.454 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.455 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.455 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.455 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.456 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.456 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.456 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.456 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.457 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.457 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754893.341581_294.1", 2020-11-19T03:06:23.457 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.458 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20", 2020-11-19T03:06:23.458 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:33.341581490Z", 2020-11-19T03:06:23.458 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.458 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.458 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.459 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.459 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.459 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_m4q0aNW1r6ONbHC6SDkHRNx", 2020-11-19T03:06:23.459 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.460 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.460 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.460 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.460 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.461 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754893.346605_295.1", 2020-11-19T03:06:23.461 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.461 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20", 2020-11-19T03:06:23.462 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:33.346605721Z", 2020-11-19T03:06:23.462 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.462 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.462 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.462 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.463 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.463 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_m4q0aNW1r6ONbHC6SDkHRNx", 2020-11-19T03:06:23.463 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754898.951630_299.1", 2020-11-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:38.951630338Z", 2020-11-19T03:06:23.466 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.466 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.466 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.466 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.467 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.467 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.467 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.467 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.468 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.468 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.468 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.469 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754898.962533_304.1", 2020-11-19T03:06:23.469 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.469 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:23.469 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:01:38.962533901Z", 2020-11-19T03:06:23.469 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.470 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.470 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.470 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.470 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.471 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.471 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.471 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.472 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.473 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754969.539612_314.1", 2020-11-19T03:06:23.473 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.473 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:23.473 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:02:49.539612219Z", 2020-11-19T03:06:23.473 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.474 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.474 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.474 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.474 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.475 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.475 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:23.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.475 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.475 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754969.549049_315.1", 2020-11-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:02:49.549049417Z", 2020-11-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.481687_339.1", 2020-11-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.481687459Z", 2020-11-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.483 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.483 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.483 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.483 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.484 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.486007_340.1", 2020-11-19T03:06:23.484 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.484 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:23.485 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.486007579Z", 2020-11-19T03:06:23.485 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.485 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.485 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.486 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.486 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.486 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:23.486 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.487 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.487 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.487 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.487 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.488 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.487864_341.1", 2020-11-19T03:06:23.488 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.488 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21", 2020-11-19T03:06:23.488 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.487864468Z", 2020-11-19T03:06:23.489 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.489 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.489 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.489 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.490 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.490 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UwTo_benAnYmaa4CegfHe2h", 2020-11-19T03:06:23.490 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.491 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.491 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.491 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.491 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.492 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.492471_342.1", 2020-11-19T03:06:23.492 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.492 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21", 2020-11-19T03:06:23.492 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.492471230Z", 2020-11-19T03:06:23.493 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.493 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.493 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.493 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.494 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.494 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UwTo_benAnYmaa4CegfHe2h", 2020-11-19T03:06:23.494 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.494 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.495 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.495 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.495 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.496 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.516426_343.1", 2020-11-19T03:06:23.496 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.496 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:23.496 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.516426936Z", 2020-11-19T03:06:23.496 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.497 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.497 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.497 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.497 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.498 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:23.498 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.498 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.498 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.499 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.499 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.499 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.520864_344.1", 2020-11-19T03:06:23.500 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.500 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:23.500 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.520864137Z", 2020-11-19T03:06:23.500 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.500 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.501 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.501 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.501 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.501 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:23.502 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.502 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.502 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.502 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.503 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.503 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.522824_345.1", 2020-11-19T03:06:23.504 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.504 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22", 2020-11-19T03:06:23.504 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.522824373Z", 2020-11-19T03:06:23.504 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.504 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.505 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.505 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.505 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.505 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-PCUgWQeHT2U7eFpqPZrnJx", 2020-11-19T03:06:23.506 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.506 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.506 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.506 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.507 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.507 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.527162_346.1", 2020-11-19T03:06:23.507 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.508 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22", 2020-11-19T03:06:23.508 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.527162988Z", 2020-11-19T03:06:23.508 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.508 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.508 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.509 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.509 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.509 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-PCUgWQeHT2U7eFpqPZrnJx", 2020-11-19T03:06:23.509 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.510 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.510 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.510 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.510 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.511 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.572422_347.1", 2020-11-19T03:06:23.511 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.511 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:23.512 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.572422879Z", 2020-11-19T03:06:23.512 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.512 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.512 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.512 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.513 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.513 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:23.513 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.578468_348.1", 2020-11-19T03:06:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.578468758Z", 2020-11-19T03:06:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.580555_349.1", 2020-11-19T03:06:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23", 2020-11-19T03:06:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.580555073Z", 2020-11-19T03:06:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_PtmtnLQARXBOkzZFBwIKFhM", 2020-11-19T03:06:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.585888_350.1", 2020-11-19T03:06:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23", 2020-11-19T03:06:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.585888316Z", 2020-11-19T03:06:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_PtmtnLQARXBOkzZFBwIKFhM", 2020-11-19T03:06:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.610407_351.1", 2020-11-19T03:06:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.610407033Z", 2020-11-19T03:06:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.616786_352.1", 2020-11-19T03:06:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.616786647Z", 2020-11-19T03:06:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.625234_353.1", 2020-11-19T03:06:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.625234108Z", 2020-11-19T03:06:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.633175_354.1", 2020-11-19T03:06:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.633175833Z", 2020-11-19T03:06:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.644998_355.1", 2020-11-19T03:06:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.644998413Z", 2020-11-19T03:06:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755009.660851_356.1", 2020-11-19T03:06:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:29.660851700Z", 2020-11-19T03:06:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.005483_361.1", 2020-11-19T03:06:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.005483628Z", 2020-11-19T03:06:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.009877_374.1", 2020-11-19T03:06:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.009877496Z", 2020-11-19T03:06:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.012023_375.1", 2020-11-19T03:06:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24", 2020-11-19T03:06:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.012023407Z", 2020-11-19T03:06:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ycH_LWaCb7pN9H3I9w2Z6N_", 2020-11-19T03:06:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.016947_376.1", 2020-11-19T03:06:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24", 2020-11-19T03:06:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.016947542Z", 2020-11-19T03:06:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ycH_LWaCb7pN9H3I9w2Z6N_", 2020-11-19T03:06:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.045357_377.1", 2020-11-19T03:06:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.045357346Z", 2020-11-19T03:06:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.050883_378.1", 2020-11-19T03:06:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.050883011Z", 2020-11-19T03:06:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.053418_379.1", 2020-11-19T03:06:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25", 2020-11-19T03:06:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.053418708Z", 2020-11-19T03:06:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bkpajY1jnfqrwJDp6NVto29", 2020-11-19T03:06:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.059396_380.1", 2020-11-19T03:06:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25", 2020-11-19T03:06:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.059396024Z", 2020-11-19T03:06:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bkpajY1jnfqrwJDp6NVto29", 2020-11-19T03:06:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.092006_381.1", 2020-11-19T03:06:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.092006149Z", 2020-11-19T03:06:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.096908_382.1", 2020-11-19T03:06:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.096908426Z", 2020-11-19T03:06:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.099132_383.1", 2020-11-19T03:06:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26", 2020-11-19T03:06:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.099132448Z", 2020-11-19T03:06:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_WAoNjTvGKPea_y2zejmRi9k", 2020-11-19T03:06:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.109731_384.1", 2020-11-19T03:06:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26", 2020-11-19T03:06:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.109731627Z", 2020-11-19T03:06:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_WAoNjTvGKPea_y2zejmRi9k", 2020-11-19T03:06:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.128366_385.1", 2020-11-19T03:06:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.128366050Z", 2020-11-19T03:06:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.134273_386.1", 2020-11-19T03:06:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.134273445Z", 2020-11-19T03:06:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.138369_387.1", 2020-11-19T03:06:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.138369812Z", 2020-11-19T03:06:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.151920_388.1", 2020-11-19T03:06:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.151920363Z", 2020-11-19T03:06:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.157598_389.1", 2020-11-19T03:06:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.157598827Z", 2020-11-19T03:06:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.167497_390.1", 2020-11-19T03:06:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.167497429Z", 2020-11-19T03:06:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.618 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.178487_391.1", 2020-11-19T03:06:23.618 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.618 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:23.618 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.178487831Z", 2020-11-19T03:06:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.186130_392.1", 2020-11-19T03:06:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.186130781Z", 2020-11-19T03:06:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.200136_393.1", 2020-11-19T03:06:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.200136493Z", 2020-11-19T03:06:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.228723_394.1", 2020-11-19T03:06:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.228723559Z", 2020-11-19T03:06:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.241089_395.1", 2020-11-19T03:06:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.241089363Z", 2020-11-19T03:06:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755021.248167_396.1", 2020-11-19T03:06:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:41.248167458Z", 2020-11-19T03:06:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 3 2020-11-19T03:06:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.070651_403.1", 2020-11-19T03:06:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.070651958Z", 2020-11-19T03:06:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.075617_416.1", 2020-11-19T03:06:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.075617906Z", 2020-11-19T03:06:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.077978_417.1", 2020-11-19T03:06:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27", 2020-11-19T03:06:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.077978681Z", 2020-11-19T03:06:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jwhmoKKel5WXgN7QZadkAxM", 2020-11-19T03:06:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.081633_418.1", 2020-11-19T03:06:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27", 2020-11-19T03:06:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.081633830Z", 2020-11-19T03:06:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jwhmoKKel5WXgN7QZadkAxM", 2020-11-19T03:06:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.106122_419.1", 2020-11-19T03:06:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.106122712Z", 2020-11-19T03:06:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.111140_420.1", 2020-11-19T03:06:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.111140679Z", 2020-11-19T03:06:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.113526_421.1", 2020-11-19T03:06:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28", 2020-11-19T03:06:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.113526370Z", 2020-11-19T03:06:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_BALtCKJ9w_4jnJuGrJpKnG-", 2020-11-19T03:06:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.118961_422.1", 2020-11-19T03:06:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28", 2020-11-19T03:06:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.118961566Z", 2020-11-19T03:06:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_BALtCKJ9w_4jnJuGrJpKnG-", 2020-11-19T03:06:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.167348_423.1", 2020-11-19T03:06:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.167348445Z", 2020-11-19T03:06:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.172591_424.1", 2020-11-19T03:06:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.172591027Z", 2020-11-19T03:06:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.174666_425.1", 2020-11-19T03:06:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29", 2020-11-19T03:06:23.680 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.174666875Z", 2020-11-19T03:06:23.680 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.680 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.680 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.680 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.681 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.681 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7NmprpCcXUiFYkFKdzDwvBW", 2020-11-19T03:06:23.681 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.682 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.682 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.683 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.178503_426.1", 2020-11-19T03:06:23.683 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.683 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29", 2020-11-19T03:06:23.683 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.178503216Z", 2020-11-19T03:06:23.684 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.684 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.684 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.684 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.685 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.685 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7NmprpCcXUiFYkFKdzDwvBW", 2020-11-19T03:06:23.685 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.685 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.686 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.686 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.686 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.687 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.200545_427.1", 2020-11-19T03:06:23.687 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.687 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:23.687 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.200545045Z", 2020-11-19T03:06:23.687 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.688 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.688 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:23.688 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.688 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.689 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:23.689 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.689 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.689 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.690 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.690 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.690 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.208326_428.1", 2020-11-19T03:06:23.691 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.691 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:23.691 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.208326306Z", 2020-11-19T03:06:23.691 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.691 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.692 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:23.692 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.692 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.692 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:23.693 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.693 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.693 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.693 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.694 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.694 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.257261_429.1", 2020-11-19T03:06:23.694 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.695 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:23.695 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.257261094Z", 2020-11-19T03:06:23.695 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.695 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.695 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:23.696 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.696 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.696 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:23.696 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.697 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.697 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.698 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.271719_430.1", 2020-11-19T03:06:23.698 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.698 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:23.699 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.271719934Z", 2020-11-19T03:06:23.699 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.699 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.699 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:23.699 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.700 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.700 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:23.700 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.700 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.701 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.701 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.701 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.702 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.348435_431.1", 2020-11-19T03:06:23.702 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.702 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:23.702 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.348435126Z", 2020-11-19T03:06:23.702 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.703 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.703 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:23.703 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.703 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.704 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:23.704 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.704 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.705 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.706 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.358783_432.1", 2020-11-19T03:06:23.706 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.706 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:23.707 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.358783093Z", 2020-11-19T03:06:23.707 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.707 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.707 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:23.708 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.708 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.709 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:23.709 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.709 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.710 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.710 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.710 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.711 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.436658_433.1", 2020-11-19T03:06:23.711 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.711 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:23.711 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.436658111Z", 2020-11-19T03:06:23.712 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.712 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.712 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.712 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.713 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.713 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:23.713 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.713 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.714 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.715 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.443801_434.1", 2020-11-19T03:06:23.715 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.715 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:23.715 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.443801075Z", 2020-11-19T03:06:23.715 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.716 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.716 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.716 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.716 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.717 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:23.717 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.717 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.717 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.718 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.447377_435.1", 2020-11-19T03:06:23.719 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.719 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30", 2020-11-19T03:06:23.719 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.447377487Z", 2020-11-19T03:06:23.719 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.719 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.720 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.720 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.720 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.720 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_mM8-TW-E_epjEOhv1WllsAD", 2020-11-19T03:06:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.452011_436.1", 2020-11-19T03:06:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30", 2020-11-19T03:06:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.452011568Z", 2020-11-19T03:06:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_mM8-TW-E_epjEOhv1WllsAD", 2020-11-19T03:06:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.486369_437.1", 2020-11-19T03:06:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.486369972Z", 2020-11-19T03:06:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.490281_438.1", 2020-11-19T03:06:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.490281477Z", 2020-11-19T03:06:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.492125_439.1", 2020-11-19T03:06:23.736 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.736 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31", 2020-11-19T03:06:23.736 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.492125730Z", 2020-11-19T03:06:23.736 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.736 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.737 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.737 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.737 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.738 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_gXleTzFvDf4A5fTuHAWH5iB", 2020-11-19T03:06:23.738 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.738 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.738 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.739 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.739 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.739 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.495843_440.1", 2020-11-19T03:06:23.740 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.740 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31", 2020-11-19T03:06:23.740 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.495843939Z", 2020-11-19T03:06:23.740 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.740 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.741 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.741 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.741 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.741 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_gXleTzFvDf4A5fTuHAWH5iB", 2020-11-19T03:06:23.742 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.742 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.742 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.742 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.743 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.743 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.528063_441.1", 2020-11-19T03:06:23.743 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.744 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:23.744 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.528063552Z", 2020-11-19T03:06:23.744 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.744 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.744 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.745 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.745 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.745 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:23.745 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.746 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.746 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.746 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.746 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.747 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.532035_442.1", 2020-11-19T03:06:23.747 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.747 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:23.748 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.532035043Z", 2020-11-19T03:06:23.748 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.748 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.748 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.748 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.749 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.749 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:23.749 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.750 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.750 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.750 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.750 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.751 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.534111_443.1", 2020-11-19T03:06:23.751 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.751 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32", 2020-11-19T03:06:23.751 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.534111357Z", 2020-11-19T03:06:23.752 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.752 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.752 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.752 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.753 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.753 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_1MkPrP26FSf3qf6aGsjMwfM", 2020-11-19T03:06:23.753 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.753 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.754 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.754 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.754 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.755 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.544040_444.1", 2020-11-19T03:06:23.755 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.755 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32", 2020-11-19T03:06:23.755 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.544040853Z", 2020-11-19T03:06:23.756 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.756 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.756 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.756 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.757 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.757 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_1MkPrP26FSf3qf6aGsjMwfM", 2020-11-19T03:06:23.757 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.757 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.758 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.758 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.758 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.759 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.575866_445.1", 2020-11-19T03:06:23.759 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.759 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:23.759 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.575866003Z", 2020-11-19T03:06:23.759 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.760 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.760 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:23.760 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.760 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.761 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:23.761 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.761 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.761 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.762 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.585216_446.1", 2020-11-19T03:06:23.763 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.763 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:23.763 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.585216241Z", 2020-11-19T03:06:23.763 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.763 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.764 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:23.764 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.764 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.764 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.639139_447.1", 2020-11-19T03:06:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.639139709Z", 2020-11-19T03:06:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.769 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.769 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.769 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.769 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.770 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.649165_448.1", 2020-11-19T03:06:23.770 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.770 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:23.771 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.649165203Z", 2020-11-19T03:06:23.771 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.771 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.771 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:23.771 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.772 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.772 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:23.772 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.773 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.773 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.773 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.773 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.710088_449.1", 2020-11-19T03:06:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.710088485Z", 2020-11-19T03:06:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.720549_450.1", 2020-11-19T03:06:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.720549878Z", 2020-11-19T03:06:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.794865_451.1", 2020-11-19T03:06:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.794865388Z", 2020-11-19T03:06:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.799032_452.1", 2020-11-19T03:06:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.799032992Z", 2020-11-19T03:06:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.801245_453.1", 2020-11-19T03:06:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33", 2020-11-19T03:06:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.801245355Z", 2020-11-19T03:06:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_8lKVoHs2FBMY1x8WlQGeuNM", 2020-11-19T03:06:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.805695_454.1", 2020-11-19T03:06:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33", 2020-11-19T03:06:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.805695695Z", 2020-11-19T03:06:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_8lKVoHs2FBMY1x8WlQGeuNM", 2020-11-19T03:06:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.832169_455.1", 2020-11-19T03:06:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.832169972Z", 2020-11-19T03:06:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.837510_456.1", 2020-11-19T03:06:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.837510588Z", 2020-11-19T03:06:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.839242_457.1", 2020-11-19T03:06:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34", 2020-11-19T03:06:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.839242953Z", 2020-11-19T03:06:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uX_C_amZ6wHaMpUOaDbLhIh", 2020-11-19T03:06:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.843268_458.1", 2020-11-19T03:06:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34", 2020-11-19T03:06:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.843268836Z", 2020-11-19T03:06:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uX_C_amZ6wHaMpUOaDbLhIh", 2020-11-19T03:06:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.918282_459.1", 2020-11-19T03:06:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.918282306Z", 2020-11-19T03:06:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.927511_460.1", 2020-11-19T03:06:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.927511062Z", 2020-11-19T03:06:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.930242_461.1", 2020-11-19T03:06:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35", 2020-11-19T03:06:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.930242283Z", 2020-11-19T03:06:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_LIlIyIKf54pQyVwNBj8qRry", 2020-11-19T03:06:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.935757_462.1", 2020-11-19T03:06:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35", 2020-11-19T03:06:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.935757929Z", 2020-11-19T03:06:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_LIlIyIKf54pQyVwNBj8qRry", 2020-11-19T03:06:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.960222_463.1", 2020-11-19T03:06:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.960222969Z", 2020-11-19T03:06:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755034.966451_464.1", 2020-11-19T03:06:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:54.966451383Z", 2020-11-19T03:06:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755036.023079_465.1", 2020-11-19T03:06:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:56.023079326Z", 2020-11-19T03:06:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755036.036438_466.1", 2020-11-19T03:06:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:56.036438127Z", 2020-11-19T03:06:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.098379_467.1", 2020-11-19T03:06:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.098379164Z", 2020-11-19T03:06:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.106961_468.1", 2020-11-19T03:06:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.106961657Z", 2020-11-19T03:06:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.187937_469.1", 2020-11-19T03:06:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36", 2020-11-19T03:06:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.187937642Z", 2020-11-19T03:06:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jRizeyH8HQBihpZqQ7H0cWx", 2020-11-19T03:06:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.191921_470.1", 2020-11-19T03:06:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36", 2020-11-19T03:06:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.191921477Z", 2020-11-19T03:06:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jRizeyH8HQBihpZqQ7H0cWx", 2020-11-19T03:06:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.193772_471.1", 2020-11-19T03:06:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-36", 2020-11-19T03:06:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.193772231Z", 2020-11-19T03:06:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_kYTvwOB6WrelYhgfkJRvPMC", 2020-11-19T03:06:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.197598_472.1", 2020-11-19T03:06:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-36", 2020-11-19T03:06:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.197598901Z", 2020-11-19T03:06:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.863 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.863 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.863 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_kYTvwOB6WrelYhgfkJRvPMC", 2020-11-19T03:06:23.863 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.864 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.864 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.864 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.864 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.865 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.235086_473.1", 2020-11-19T03:06:23.865 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.866 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37", 2020-11-19T03:06:23.866 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.235086410Z", 2020-11-19T03:06:23.866 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.866 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.866 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.867 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.867 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.867 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_my5SPqbG7dQQSEBcU42t2-N", 2020-11-19T03:06:23.867 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.868 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.868 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.868 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.868 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.869 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.239751_474.1", 2020-11-19T03:06:23.869 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.869 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37", 2020-11-19T03:06:23.870 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.239751394Z", 2020-11-19T03:06:23.870 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.870 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.870 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.870 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.871 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.871 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_my5SPqbG7dQQSEBcU42t2-N", 2020-11-19T03:06:23.871 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.872 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.872 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.872 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.872 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.873 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.242234_475.1", 2020-11-19T03:06:23.873 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.873 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-37", 2020-11-19T03:06:23.874 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.242234716Z", 2020-11-19T03:06:23.874 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.874 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.874 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.874 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.875 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.875 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_g91x9CrlSKmRRavm4MwY2r5", 2020-11-19T03:06:23.875 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.875 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.876 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.877 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755037.247176_476.1", 2020-11-19T03:06:23.877 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.877 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-37", 2020-11-19T03:06:23.877 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:57.247176520Z", 2020-11-19T03:06:23.878 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.878 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.878 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.878 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.879 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.879 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_g91x9CrlSKmRRavm4MwY2r5", 2020-11-19T03:06:23.879 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.879 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.880 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.880 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755038.595496_477.1", 2020-11-19T03:06:23.881 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.881 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38", 2020-11-19T03:06:23.881 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:58.595496690Z", 2020-11-19T03:06:23.881 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.881 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.882 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.882 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.882 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.882 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V7n32juggP5QmeLeOGxy3UK", 2020-11-19T03:06:23.883 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.883 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.883 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.883 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.884 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755038.599524_478.1", 2020-11-19T03:06:23.884 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.885 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38", 2020-11-19T03:06:23.885 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:58.599524203Z", 2020-11-19T03:06:23.885 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.885 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.885 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.886 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.886 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.886 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V7n32juggP5QmeLeOGxy3UK", 2020-11-19T03:06:23.886 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.887 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.887 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.887 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.887 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.888 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755038.601388_479.1", 2020-11-19T03:06:23.888 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.888 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-38", 2020-11-19T03:06:23.889 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:58.601388404Z", 2020-11-19T03:06:23.889 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.889 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.889 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.889 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.890 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.890 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_W1ZCfu17NUxLx7oS231exeH", 2020-11-19T03:06:23.890 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.891 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.891 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.891 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.891 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.892 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755038.605369_480.1", 2020-11-19T03:06:23.892 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.892 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-38", 2020-11-19T03:06:23.892 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:03:58.605369575Z", 2020-11-19T03:06:23.893 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.893 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.893 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.893 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.894 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.894 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_W1ZCfu17NUxLx7oS231exeH", 2020-11-19T03:06:23.894 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.894 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.895 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.895 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.895 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.896 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.318171_485.1", 2020-11-19T03:06:23.896 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.896 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-39:d71482d6-15fd-4715-9992-a25958dcb538.4341.39", 2020-11-19T03:06:23.896 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.318171704Z", 2020-11-19T03:06:23.896 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.897 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.897 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.897 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.897 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.898 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MR6DDMoW-bWzeQ2lVYjkv_T", 2020-11-19T03:06:23.898 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.898 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.898 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.899 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.899 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.899 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.326000_486.1", 2020-11-19T03:06:23.900 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.900 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-39:d71482d6-15fd-4715-9992-a25958dcb538.4341.39", 2020-11-19T03:06:23.900 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.326000603Z", 2020-11-19T03:06:23.900 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.900 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.901 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.901 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.901 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.901 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_MR6DDMoW-bWzeQ2lVYjkv_T", 2020-11-19T03:06:23.902 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.902 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.902 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.902 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.903 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.903 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.328429_487.1", 2020-11-19T03:06:23.903 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.904 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-39", 2020-11-19T03:06:23.904 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.328429735Z", 2020-11-19T03:06:23.904 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.904 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.904 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.905 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.905 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.905 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2xs16O_Y3iMqtaOm0GzhjDK", 2020-11-19T03:06:23.905 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.906 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.906 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.906 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.906 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.907 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.333402_488.1", 2020-11-19T03:06:23.907 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.907 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-39", 2020-11-19T03:06:23.908 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.333402182Z", 2020-11-19T03:06:23.908 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.908 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.908 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.908 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.909 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.909 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2xs16O_Y3iMqtaOm0GzhjDK", 2020-11-19T03:06:23.909 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.910 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.910 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.910 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.910 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.911 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.366052_489.1", 2020-11-19T03:06:23.911 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.911 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-40:d71482d6-15fd-4715-9992-a25958dcb538.4341.40", 2020-11-19T03:06:23.911 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.366052352Z", 2020-11-19T03:06:23.912 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.912 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.912 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.912 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.913 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.913 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_oGE_O83RX1iSJnFe2p4H0J-", 2020-11-19T03:06:23.913 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.913 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.914 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.915 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.372774_490.1", 2020-11-19T03:06:23.915 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.915 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-40:d71482d6-15fd-4715-9992-a25958dcb538.4341.40", 2020-11-19T03:06:23.915 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.372774679Z", 2020-11-19T03:06:23.915 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.916 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.916 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.916 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.916 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.917 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_oGE_O83RX1iSJnFe2p4H0J-", 2020-11-19T03:06:23.917 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.917 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.917 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.918 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.918 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.918 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.375737_491.1", 2020-11-19T03:06:23.919 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.919 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-40", 2020-11-19T03:06:23.919 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.375737830Z", 2020-11-19T03:06:23.919 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.919 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.920 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.920 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.920 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.920 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bGn8OsdsLasWP-8EeCEuz7n", 2020-11-19T03:06:23.921 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.921 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.921 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.921 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.922 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.922 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.382371_492.1", 2020-11-19T03:06:23.922 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-40", 2020-11-19T03:06:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.382371633Z", 2020-11-19T03:06:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bGn8OsdsLasWP-8EeCEuz7n", 2020-11-19T03:06:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.429548_493.1", 2020-11-19T03:06:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-41:d71482d6-15fd-4715-9992-a25958dcb538.4341.41", 2020-11-19T03:06:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.429548543Z", 2020-11-19T03:06:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.928 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.928 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__wSJMtQIou6xYhPPtSS8YuA", 2020-11-19T03:06:23.928 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.929 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.929 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.930 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.436987_494.1", 2020-11-19T03:06:23.930 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.930 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-41:d71482d6-15fd-4715-9992-a25958dcb538.4341.41", 2020-11-19T03:06:23.931 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.436987958Z", 2020-11-19T03:06:23.931 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.931 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.931 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.931 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.932 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.932 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__wSJMtQIou6xYhPPtSS8YuA", 2020-11-19T03:06:23.932 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.933 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.933 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.933 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.933 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.934 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.439804_495.1", 2020-11-19T03:06:23.934 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.934 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-41", 2020-11-19T03:06:23.934 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.439804366Z", 2020-11-19T03:06:23.935 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.935 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.935 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.935 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.936 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.936 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_RsdLOBWehIFSw4jrAQphm1V", 2020-11-19T03:06:23.936 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.936 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.937 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.937 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.937 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.938 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755040.443579_496.1", 2020-11-19T03:06:23.938 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.938 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-41", 2020-11-19T03:06:23.938 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.443579496Z", 2020-11-19T03:06:23.938 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.939 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.939 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.939 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.939 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.940 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_RsdLOBWehIFSw4jrAQphm1V", 2020-11-19T03:06:23.940 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.940 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.940 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.941 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.941 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.941 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.049424_71.1", 2020-11-19T03:06:23.942 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.942 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-1:d71482d6-15fd-4715-9992-a25958dcb538.4341.1", 2020-11-19T03:06:23.942 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.049424391Z", 2020-11-19T03:06:23.942 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.942 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.943 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.943 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.943 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.943 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qgItci47XzVPPEUbRm9FP71", 2020-11-19T03:06:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.944 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.067007_75.1", 2020-11-19T03:06:23.945 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-1:d71482d6-15fd-4715-9992-a25958dcb538.4341.1", 2020-11-19T03:06:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.067007384Z", 2020-11-19T03:06:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.946 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.947 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.947 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.947 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qgItci47XzVPPEUbRm9FP71", 2020-11-19T03:06:23.947 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.948 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.948 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.948 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.948 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.949 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.068631_76.1", 2020-11-19T03:06:23.949 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.949 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-1", 2020-11-19T03:06:23.950 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.068631964Z", 2020-11-19T03:06:23.950 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.950 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.950 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.950 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.951 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.951 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_94mQ_WxhbwDk-kFSjKqwmYF", 2020-11-19T03:06:23.951 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.071728_77.1", 2020-11-19T03:06:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-1", 2020-11-19T03:06:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.071728952Z", 2020-11-19T03:06:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_94mQ_WxhbwDk-kFSjKqwmYF", 2020-11-19T03:06:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.094013_78.1", 2020-11-19T03:06:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-2:d71482d6-15fd-4715-9992-a25958dcb538.4341.2", 2020-11-19T03:06:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.094013583Z", 2020-11-19T03:06:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bSYBD3JTM-OkT97a99BJ1cL", 2020-11-19T03:06:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.098511_79.1", 2020-11-19T03:06:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-2:d71482d6-15fd-4715-9992-a25958dcb538.4341.2", 2020-11-19T03:06:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.098511431Z", 2020-11-19T03:06:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_bSYBD3JTM-OkT97a99BJ1cL", 2020-11-19T03:06:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.100851_80.1", 2020-11-19T03:06:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-2", 2020-11-19T03:06:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.100851568Z", 2020-11-19T03:06:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.966 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.966 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.966 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5UuoR0SlRK-wajXJjy8RaHm", 2020-11-19T03:06:23.966 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.967 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.967 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.967 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.967 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.968 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754535.105586_81.1", 2020-11-19T03:06:23.968 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.968 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-2", 2020-11-19T03:06:23.969 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:35.105586168Z", 2020-11-19T03:06:23.969 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.969 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.969 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.969 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.970 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.970 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5UuoR0SlRK-wajXJjy8RaHm", 2020-11-19T03:06:23.970 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.971 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.971 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.972 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754538.238498_82.1", 2020-11-19T03:06:23.972 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.972 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-3:d71482d6-15fd-4715-9992-a25958dcb538.4341.3", 2020-11-19T03:06:23.973 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:38.238498971Z", 2020-11-19T03:06:23.973 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.973 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.973 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.973 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.974 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.974 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_GAW34u3sdJwjuChdK6wHKmm", 2020-11-19T03:06:23.974 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.975 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.975 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.975 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.975 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.976 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754538.263102_83.1", 2020-11-19T03:06:23.976 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.976 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-3:d71482d6-15fd-4715-9992-a25958dcb538.4341.3", 2020-11-19T03:06:23.976 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:38.263102074Z", 2020-11-19T03:06:23.977 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.977 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.977 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.977 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.978 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.978 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_GAW34u3sdJwjuChdK6wHKmm", 2020-11-19T03:06:23.978 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.978 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.979 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.979 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.979 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.980 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754538.265151_84.1", 2020-11-19T03:06:23.980 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.980 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-3", 2020-11-19T03:06:23.980 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:38.265151277Z", 2020-11-19T03:06:23.980 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.981 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.981 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.981 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.981 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.982 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_tL7nc-Jhsvjv2EEZRXT73OW", 2020-11-19T03:06:23.982 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.982 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.982 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.983 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754538.270230_85.1", 2020-11-19T03:06:23.984 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.984 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-3", 2020-11-19T03:06:23.984 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:38.270230968Z", 2020-11-19T03:06:23.984 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.984 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.985 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.985 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.985 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.985 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_tL7nc-Jhsvjv2EEZRXT73OW", 2020-11-19T03:06:23.986 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.986 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.986 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.986 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.987 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.987 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.646238_86.1", 2020-11-19T03:06:23.987 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.988 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-4:d71482d6-15fd-4715-9992-a25958dcb538.4341.4", 2020-11-19T03:06:23.988 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.646238073Z", 2020-11-19T03:06:23.988 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.988 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.988 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.989 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.989 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.989 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_kXyu9LGpqsO6w8QSZAhud3g", 2020-11-19T03:06:23.989 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.990 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.990 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.991 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.653311_88.1", 2020-11-19T03:06:23.991 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:23.991 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-4:d71482d6-15fd-4715-9992-a25958dcb538.4341.4", 2020-11-19T03:06:23.992 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.653311568Z", 2020-11-19T03:06:23.992 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.992 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.992 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.992 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.993 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.993 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_kXyu9LGpqsO6w8QSZAhud3g", 2020-11-19T03:06:23.993 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.993 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.994 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:23.994 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.994 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.995 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.655601_89.1", 2020-11-19T03:06:23.995 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.995 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-4", 2020-11-19T03:06:23.995 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.655601491Z", 2020-11-19T03:06:23.995 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.996 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:23.996 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:23.996 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:23.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.996 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:23.997 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_tMmjLsHDWqdVdqcqT0YXzke", 2020-11-19T03:06:23.997 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:23.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.997 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:23.997 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:23.998 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.998 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:23.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:23.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:23.998 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.661306_90.1", 2020-11-19T03:06:23.999 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:23.999 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-4", 2020-11-19T03:06:23.999 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.661306950Z", 2020-11-19T03:06:23.999 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:23.999 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.000 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.000 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.000 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.000 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_tMmjLsHDWqdVdqcqT0YXzke", 2020-11-19T03:06:24.001 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.001 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.001 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.002 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.002 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.002 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.685988_91.1", 2020-11-19T03:06:24.003 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.003 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-5:d71482d6-15fd-4715-9992-a25958dcb538.4341.5", 2020-11-19T03:06:24.003 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.685988313Z", 2020-11-19T03:06:24.003 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.003 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.004 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.004 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.004 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.004 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qVWvmOqSOj4nJ_IyDQjqQEP", 2020-11-19T03:06:24.005 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.005 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.005 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.005 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.006 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.006 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.690669_92.1", 2020-11-19T03:06:24.006 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.007 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-5:d71482d6-15fd-4715-9992-a25958dcb538.4341.5", 2020-11-19T03:06:24.007 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.690669131Z", 2020-11-19T03:06:24.007 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.007 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.007 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.008 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.008 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.008 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qVWvmOqSOj4nJ_IyDQjqQEP", 2020-11-19T03:06:24.008 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.009 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.009 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.009 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.009 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.010 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.693071_93.1", 2020-11-19T03:06:24.010 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.010 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-5", 2020-11-19T03:06:24.011 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.693071890Z", 2020-11-19T03:06:24.011 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.011 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.011 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.011 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.012 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.012 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_IRRwWfE5ELTX6B9PeD0Wss2", 2020-11-19T03:06:24.012 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.013 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.013 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.014 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.697214_94.1", 2020-11-19T03:06:24.014 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.014 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-5", 2020-11-19T03:06:24.014 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.697214643Z", 2020-11-19T03:06:24.015 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.015 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.015 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.015 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.016 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.016 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_IRRwWfE5ELTX6B9PeD0Wss2", 2020-11-19T03:06:24.016 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.016 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.017 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.017 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.017 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.018 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.740644_95.1", 2020-11-19T03:06:24.018 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.018 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-6:d71482d6-15fd-4715-9992-a25958dcb538.4341.6", 2020-11-19T03:06:24.018 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.740644196Z", 2020-11-19T03:06:24.018 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.019 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.019 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.019 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.019 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.020 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qrn-62ONHSx8esDlsD49N2S", 2020-11-19T03:06:24.020 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.020 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.020 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.021 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.744532_96.1", 2020-11-19T03:06:24.022 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.022 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-6:d71482d6-15fd-4715-9992-a25958dcb538.4341.6", 2020-11-19T03:06:24.022 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.744532390Z", 2020-11-19T03:06:24.022 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.022 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.023 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.023 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.023 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.023 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_qrn-62ONHSx8esDlsD49N2S", 2020-11-19T03:06:24.024 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.024 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.024 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.024 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.025 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.025 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.746440_97.1", 2020-11-19T03:06:24.025 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.026 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-6", 2020-11-19T03:06:24.026 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.746440476Z", 2020-11-19T03:06:24.026 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.026 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.026 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.027 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.027 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.027 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_wFarfv5IWniL3C-MOrXA7X0", 2020-11-19T03:06:24.027 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.028 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.028 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.028 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.028 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.029 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754539.750854_98.1", 2020-11-19T03:06:24.029 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.029 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-6", 2020-11-19T03:06:24.030 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:39.750854665Z", 2020-11-19T03:06:24.030 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.030 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.030 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.030 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_wFarfv5IWniL3C-MOrXA7X0", 2020-11-19T03:06:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.033 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.678670_99.1", 2020-11-19T03:06:24.033 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.033 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7:d71482d6-15fd-4715-9992-a25958dcb538.4341.7", 2020-11-19T03:06:24.034 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.678670320Z", 2020-11-19T03:06:24.034 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.034 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.034 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.034 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.035 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.035 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3L9q5SfSgUsZej9VYTkemdm", 2020-11-19T03:06:24.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.684139_102.1", 2020-11-19T03:06:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7:d71482d6-15fd-4715-9992-a25958dcb538.4341.7", 2020-11-19T03:06:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.684139704Z", 2020-11-19T03:06:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_3L9q5SfSgUsZej9VYTkemdm", 2020-11-19T03:06:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.040 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.686054_103.1", 2020-11-19T03:06:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7", 2020-11-19T03:06:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.686054368Z", 2020-11-19T03:06:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.690172_104.1", 2020-11-19T03:06:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7", 2020-11-19T03:06:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.690172550Z", 2020-11-19T03:06:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.047 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:24.047 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.047 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.047 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.047 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.048 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.048 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.714743_105.1", 2020-11-19T03:06:24.048 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8:d71482d6-15fd-4715-9992-a25958dcb538.4341.8", 2020-11-19T03:06:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.714743025Z", 2020-11-19T03:06:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Y2tOicXm39NSETUiNVpNZLc", 2020-11-19T03:06:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.052 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.052 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.719653_106.1", 2020-11-19T03:06:24.052 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.053 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8:d71482d6-15fd-4715-9992-a25958dcb538.4341.8", 2020-11-19T03:06:24.053 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.719653546Z", 2020-11-19T03:06:24.053 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.053 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.053 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_Y2tOicXm39NSETUiNVpNZLc", 2020-11-19T03:06:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.055 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.055 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.056 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.722344_107.1", 2020-11-19T03:06:24.056 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.056 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8", 2020-11-19T03:06:24.057 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.722344069Z", 2020-11-19T03:06:24.057 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.057 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.057 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.057 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.058 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.058 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:24.058 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.727853_108.1", 2020-11-19T03:06:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8", 2020-11-19T03:06:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.727853163Z", 2020-11-19T03:06:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.773887_109.1", 2020-11-19T03:06:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9:d71482d6-15fd-4715-9992-a25958dcb538.4341.9", 2020-11-19T03:06:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.773887989Z", 2020-11-19T03:06:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.066 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_vGJjA-wwP6rD2bH8RFgbNPS", 2020-11-19T03:06:24.066 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.066 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.066 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.067 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.067 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.067 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.778690_110.1", 2020-11-19T03:06:24.067 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9:d71482d6-15fd-4715-9992-a25958dcb538.4341.9", 2020-11-19T03:06:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.778690126Z", 2020-11-19T03:06:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_vGJjA-wwP6rD2bH8RFgbNPS", 2020-11-19T03:06:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.781507_111.1", 2020-11-19T03:06:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9", 2020-11-19T03:06:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.781507411Z", 2020-11-19T03:06:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.074 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.074 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:24.074 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.074 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.075 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754542.787262_112.1", 2020-11-19T03:06:24.075 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.075 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9", 2020-11-19T03:06:24.075 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:42.787262520Z", 2020-11-19T03:06:24.076 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.076 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.076 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:24.076 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:24.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.077 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.077 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:24.077 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.078 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.078 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.078 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.078 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.079 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.124200_113.1", 2020-11-19T03:06:24.079 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.079 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7", 2020-11-19T03:06:24.079 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.124200729Z", 2020-11-19T03:06:24.080 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.080 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.080 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:24.080 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.081 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.081 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:24.081 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:24.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.081 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.082 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "remove" 2020-11-19T03:06:24.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.083 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.128855_115.1", 2020-11-19T03:06:24.083 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.083 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-7", 2020-11-19T03:06:24.083 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.128855513Z", 2020-11-19T03:06:24.083 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.083 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.084 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:24.084 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.084 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.084 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:24.085 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:24.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.085 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.085 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.085 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.086 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.086 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.183608_116.1", 2020-11-19T03:06:24.086 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.087 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8", 2020-11-19T03:06:24.087 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.183608180Z", 2020-11-19T03:06:24.087 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.087 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.087 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:24.088 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.088 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.088 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:24.088 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:24.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.089 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.089 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "remove" 2020-11-19T03:06:24.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.090 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.189052_117.1", 2020-11-19T03:06:24.090 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.090 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-8", 2020-11-19T03:06:24.091 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.189052423Z", 2020-11-19T03:06:24.091 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.091 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.091 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:24.091 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.092 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.092 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:24.092 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:24.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.093 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.093 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.093 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.093 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.094 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.288556_118.1", 2020-11-19T03:06:24.094 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.094 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9", 2020-11-19T03:06:24.094 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.288556017Z", 2020-11-19T03:06:24.095 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.095 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.095 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:24.095 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.096 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.096 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:24.096 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:24.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.096 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.097 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "remove" 2020-11-19T03:06:24.097 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.097 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:24.098 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605754544.294580_119.1", 2020-11-19T03:06:24.098 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:24.098 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-9", 2020-11-19T03:06:24.098 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580366Z", 2020-11-19T03:06:24.098 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:24.099 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:24.099 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:24.099 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:24.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.099 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:24.100 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:24.100 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 2 2020-11-19T03:06:24.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:24.100 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:24.100 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:24.101 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.101 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.101 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:24.101 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:24.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:24.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:24.104 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b2c8afed-318c-484c-8dd4-daab7a757391 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:24.144 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:24.145 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:24.374 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:06:24.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.375 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754414.632290_106.1", 2020-11-19T03:06:24.375 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "user", 2020-11-19T03:06:24.376 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "rgw-multisite-test-user", 2020-11-19T03:06:24.376 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.632290363Z", 2020-11-19T03:06:24.376 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.377 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.377 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.377 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.378 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.378 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OD787OEqCb37r-dWrIu2W6g", 2020-11-19T03:06:24.378 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.379 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.379 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.379 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.379 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.380 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754414.640760_109.1", 2020-11-19T03:06:24.380 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "user", 2020-11-19T03:06:24.380 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "rgw-multisite-test-user", 2020-11-19T03:06:24.381 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:53:34.640760721Z", 2020-11-19T03:06:24.381 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.381 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.381 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.382 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.382 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.382 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OD787OEqCb37r-dWrIu2W6g", 2020-11-19T03:06:24.383 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.383 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.383 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.384 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.384 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.384 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.755439_114.1", 2020-11-19T03:06:24.385 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.385 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10", 2020-11-19T03:06:24.385 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.755439437Z", 2020-11-19T03:06:24.385 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.386 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.386 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.386 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.387 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.387 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jDQMSP1imZk9M53g_LemGsD", 2020-11-19T03:06:24.387 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.388 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.388 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.388 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.388 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.389 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.759756_127.1", 2020-11-19T03:06:24.389 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.389 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10", 2020-11-19T03:06:24.390 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.759756660Z", 2020-11-19T03:06:24.390 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.390 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.390 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.390 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.391 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.391 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jDQMSP1imZk9M53g_LemGsD", 2020-11-19T03:06:24.391 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.392 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.392 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.392 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.393 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.393 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.761697_128.1", 2020-11-19T03:06:24.393 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.394 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-10", 2020-11-19T03:06:24.394 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.761697955Z", 2020-11-19T03:06:24.394 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.394 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.395 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.395 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.395 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.396 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_QaR_8xY7tB3j-5_fwfgIAp2", 2020-11-19T03:06:24.396 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.396 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.397 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.397 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.397 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.398 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.765274_129.1", 2020-11-19T03:06:24.398 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.398 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-10", 2020-11-19T03:06:24.399 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.765274847Z", 2020-11-19T03:06:24.399 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.399 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.400 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.400 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.400 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.401 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_QaR_8xY7tB3j-5_fwfgIAp2", 2020-11-19T03:06:24.402 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.402 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.403 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.403 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.403 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.404 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.789431_130.1", 2020-11-19T03:06:24.404 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.404 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11", 2020-11-19T03:06:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.789431860Z", 2020-11-19T03:06:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_h4zkJ0TD5OOIK8n5LNF_3YA", 2020-11-19T03:06:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.793774_131.1", 2020-11-19T03:06:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.409 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11", 2020-11-19T03:06:24.409 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.793774959Z", 2020-11-19T03:06:24.409 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.409 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.410 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.410 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.410 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.411 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_h4zkJ0TD5OOIK8n5LNF_3YA", 2020-11-19T03:06:24.411 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.411 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.412 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.412 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.412 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.413 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.798353_132.1", 2020-11-19T03:06:24.413 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.413 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-11", 2020-11-19T03:06:24.414 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.798353113Z", 2020-11-19T03:06:24.414 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.414 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.414 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.415 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.415 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.415 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5lbvhy-2ssHalMo0XAyb9jK", 2020-11-19T03:06:24.415 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.416 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.416 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.416 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.417 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.417 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.803735_133.1", 2020-11-19T03:06:24.418 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.418 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-11", 2020-11-19T03:06:24.418 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.803735217Z", 2020-11-19T03:06:24.418 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.418 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.419 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.419 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.419 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.420 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5lbvhy-2ssHalMo0XAyb9jK", 2020-11-19T03:06:24.420 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.420 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.420 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.421 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.421 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.422 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.854454_134.1", 2020-11-19T03:06:24.422 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.422 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12", 2020-11-19T03:06:24.423 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.854454685Z", 2020-11-19T03:06:24.423 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.423 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.423 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.424 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.424 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.424 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_DG8ZKomIxnXa3vMXcvP8ajO", 2020-11-19T03:06:24.425 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.425 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.425 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.426 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.426 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.426 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.427 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.858973_135.1", 2020-11-19T03:06:24.427 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.427 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12", 2020-11-19T03:06:24.427 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.858973052Z", 2020-11-19T03:06:24.428 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.428 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.428 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.428 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.429 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.429 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_DG8ZKomIxnXa3vMXcvP8ajO", 2020-11-19T03:06:24.429 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.430 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.430 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.430 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.431 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.431 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.860848_136.1", 2020-11-19T03:06:24.432 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.432 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-12", 2020-11-19T03:06:24.432 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.860848274Z", 2020-11-19T03:06:24.433 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.433 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.433 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.433 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.434 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.434 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xHh-uu7OMNHE3E1tkdnCoN0", 2020-11-19T03:06:24.434 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.435 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.435 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.435 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.436 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.436 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754545.866167_137.1", 2020-11-19T03:06:24.436 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.437 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-12", 2020-11-19T03:06:24.437 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:45.866167085Z", 2020-11-19T03:06:24.437 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.437 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.438 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.438 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.438 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xHh-uu7OMNHE3E1tkdnCoN0", 2020-11-19T03:06:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.440 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.440 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.440 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.441 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.787642_145.1", 2020-11-19T03:06:24.441 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.441 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13", 2020-11-19T03:06:24.441 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.787642625Z", 2020-11-19T03:06:24.442 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.442 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.442 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.443 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.443 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.443 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qxTzHkbF9wemeZVynRNqeCj", 2020-11-19T03:06:24.444 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.444 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.444 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.444 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.445 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.445 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.792198_150.1", 2020-11-19T03:06:24.446 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.446 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13", 2020-11-19T03:06:24.446 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.792198045Z", 2020-11-19T03:06:24.447 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.447 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.447 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.447 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.448 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.448 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qxTzHkbF9wemeZVynRNqeCj", 2020-11-19T03:06:24.448 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.449 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.449 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.449 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.794166_151.1", 2020-11-19T03:06:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.451 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-13", 2020-11-19T03:06:24.451 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.794166672Z", 2020-11-19T03:06:24.451 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.451 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.452 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.452 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.452 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.453 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ogTrNysaHhYexwepXbm3dwI", 2020-11-19T03:06:24.453 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.454 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.454 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.454 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.454 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.455 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.798068_152.1", 2020-11-19T03:06:24.455 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.456 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-13", 2020-11-19T03:06:24.456 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.798068026Z", 2020-11-19T03:06:24.456 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.456 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.457 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.457 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.457 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.457 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ogTrNysaHhYexwepXbm3dwI", 2020-11-19T03:06:24.458 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.458 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.458 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.459 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.459 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.459 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.819538_154.1", 2020-11-19T03:06:24.460 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.460 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14", 2020-11-19T03:06:24.460 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.819538205Z", 2020-11-19T03:06:24.460 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.461 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.461 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.461 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.462 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.462 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dcIW3-alzbWV6sgq13MRCG7", 2020-11-19T03:06:24.462 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.463 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.463 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.463 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.464 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.464 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.825038_155.1", 2020-11-19T03:06:24.465 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.465 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14", 2020-11-19T03:06:24.465 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.825038088Z", 2020-11-19T03:06:24.465 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.466 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.466 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.466 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.467 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.467 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dcIW3-alzbWV6sgq13MRCG7", 2020-11-19T03:06:24.467 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.468 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.468 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.468 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.468 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.469 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.827341_156.1", 2020-11-19T03:06:24.469 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.469 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-14", 2020-11-19T03:06:24.470 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.827341897Z", 2020-11-19T03:06:24.470 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.470 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.470 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.471 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.471 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.471 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_TbVBTXwBuNJEI3zmckRRIjz", 2020-11-19T03:06:24.472 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.472 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.472 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.474 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.832699_157.1", 2020-11-19T03:06:24.474 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.474 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-14", 2020-11-19T03:06:24.474 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.832699383Z", 2020-11-19T03:06:24.475 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.475 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.475 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.475 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.476 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.476 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_TbVBTXwBuNJEI3zmckRRIjz", 2020-11-19T03:06:24.476 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.477 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.477 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.477 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.477 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.478 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.868186_158.1", 2020-11-19T03:06:24.478 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.479 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15", 2020-11-19T03:06:24.479 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.868186115Z", 2020-11-19T03:06:24.479 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.479 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.480 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.480 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.480 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Pzb_ZFMDCfzlSj1lsKyhj_z", 2020-11-19T03:06:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.482 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.482 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.482 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.483 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.872546_159.1", 2020-11-19T03:06:24.483 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.483 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15", 2020-11-19T03:06:24.483 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.872546446Z", 2020-11-19T03:06:24.484 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.484 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.484 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.484 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.485 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.485 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Pzb_ZFMDCfzlSj1lsKyhj_z", 2020-11-19T03:06:24.486 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.486 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.486 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.488 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.876242_160.1", 2020-11-19T03:06:24.488 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.488 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-15", 2020-11-19T03:06:24.488 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.876242156Z", 2020-11-19T03:06:24.489 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.489 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.489 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.489 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.490 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.490 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Hkg8dq-5IxwuJ-1F2ioM3Ha", 2020-11-19T03:06:24.490 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.491 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.491 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.492 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754557.888825_161.1", 2020-11-19T03:06:24.493 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.493 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-15", 2020-11-19T03:06:24.493 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:57.888825992Z", 2020-11-19T03:06:24.493 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.493 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.494 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.494 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.494 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.495 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Hkg8dq-5IxwuJ-1F2ioM3Ha", 2020-11-19T03:06:24.495 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.495 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.496 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.496 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.496 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.497 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.798543_207.1", 2020-11-19T03:06:24.497 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.497 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:24.497 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.798543679Z", 2020-11-19T03:06:24.498 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.498 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.498 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.498 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.499 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.499 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:24.499 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.500 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.500 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.500 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.501 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.501 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.803204_211.1", 2020-11-19T03:06:24.502 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.502 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:24.502 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.803204912Z", 2020-11-19T03:06:24.502 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.503 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.503 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.503 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.503 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.504 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:24.504 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.504 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.505 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.505 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.505 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.506 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.805294_212.1", 2020-11-19T03:06:24.506 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.506 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16", 2020-11-19T03:06:24.507 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.805294677Z", 2020-11-19T03:06:24.507 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.507 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.507 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.508 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.508 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.508 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8SdF_vsimBqHGDaqiMgkarR", 2020-11-19T03:06:24.509 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.509 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.509 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.510 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.810747_213.1", 2020-11-19T03:06:24.511 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.511 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16", 2020-11-19T03:06:24.511 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.810747502Z", 2020-11-19T03:06:24.511 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.512 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.512 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.512 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.513 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.513 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8SdF_vsimBqHGDaqiMgkarR", 2020-11-19T03:06:24.513 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.514 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.514 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.515 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.838151_214.1", 2020-11-19T03:06:24.515 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.516 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:24.516 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.838151294Z", 2020-11-19T03:06:24.516 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.517 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.517 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.517 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.518 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.518 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:24.518 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.518 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.519 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.520 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.842707_215.1", 2020-11-19T03:06:24.520 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.520 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:24.520 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.842707380Z", 2020-11-19T03:06:24.521 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.521 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.521 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.521 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.522 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.522 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:24.522 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.523 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.523 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.523 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.524 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.525 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.844853_216.1", 2020-11-19T03:06:24.525 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.525 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17", 2020-11-19T03:06:24.525 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.844853013Z", 2020-11-19T03:06:24.526 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.526 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.526 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.526 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.527 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.527 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__nKT7kdBFED-s1dk1lgSXVH", 2020-11-19T03:06:24.527 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.528 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.528 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.528 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.529 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.529 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.849738_217.1", 2020-11-19T03:06:24.530 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.530 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17", 2020-11-19T03:06:24.530 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.849738526Z", 2020-11-19T03:06:24.530 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.531 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.531 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.531 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.532 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.532 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__nKT7kdBFED-s1dk1lgSXVH", 2020-11-19T03:06:24.532 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.533 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.533 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.533 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.533 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.534 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.905621_218.1", 2020-11-19T03:06:24.534 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.535 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:24.535 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.905621496Z", 2020-11-19T03:06:24.535 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.536 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.536 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.536 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.537 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.537 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:24.537 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.538 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.538 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.539 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.911693_219.1", 2020-11-19T03:06:24.539 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.539 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:24.540 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.911693551Z", 2020-11-19T03:06:24.540 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.540 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.540 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.541 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.541 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.541 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:24.542 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.542 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.542 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.543 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.543 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.543 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.915252_220.1", 2020-11-19T03:06:24.544 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.544 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18", 2020-11-19T03:06:24.544 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.915252896Z", 2020-11-19T03:06:24.545 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.545 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.545 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.545 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.546 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.546 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6l8yn5mDyDSfq7DcU_4ezuR", 2020-11-19T03:06:24.546 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.547 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.547 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.548 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.919671_221.1", 2020-11-19T03:06:24.549 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.549 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18", 2020-11-19T03:06:24.549 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.919671168Z", 2020-11-19T03:06:24.549 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.549 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.550 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.550 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.550 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.551 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6l8yn5mDyDSfq7DcU_4ezuR", 2020-11-19T03:06:24.551 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.551 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.551 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.553 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.939873_222.1", 2020-11-19T03:06:24.553 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.553 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:24.553 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.939873846Z", 2020-11-19T03:06:24.554 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.554 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.554 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:24.554 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.555 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.555 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:24.556 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.556 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.556 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.558 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.948027_223.1", 2020-11-19T03:06:24.558 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.558 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16", 2020-11-19T03:06:24.558 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.948027995Z", 2020-11-19T03:06:24.559 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.559 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.559 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:24.559 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.560 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.560 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fnvvGZ0d7lfLrS3FGGJswLo", 2020-11-19T03:06:24.560 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.561 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.561 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.561 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.562 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.562 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.955102_224.1", 2020-11-19T03:06:24.562 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.563 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:24.563 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.955102556Z", 2020-11-19T03:06:24.563 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.563 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.564 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:24.564 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.564 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.565 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:24.565 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.566 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.566 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.567 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.962377_225.1", 2020-11-19T03:06:24.567 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.568 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17", 2020-11-19T03:06:24.568 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.962377954Z", 2020-11-19T03:06:24.568 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.568 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.569 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:24.569 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.569 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.569 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_s96UHC1FXk2Dbh3d-FczUq2", 2020-11-19T03:06:24.570 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.570 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.570 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.572 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.974309_226.1", 2020-11-19T03:06:24.572 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.572 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:24.572 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.974309142Z", 2020-11-19T03:06:24.573 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.573 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.573 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:24.573 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.574 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.574 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:24.574 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.575 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.575 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.576 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754607.983290_227.1", 2020-11-19T03:06:24.576 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.577 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18", 2020-11-19T03:06:24.577 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:56:47.983290859Z", 2020-11-19T03:06:24.577 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.577 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.578 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:24.578 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.578 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.579 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p6PV58hMcYCjMu93mvIJdBa", 2020-11-19T03:06:24.579 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.579 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.580 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.580 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.580 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.581 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.853615_253.1", 2020-11-19T03:06:24.581 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.581 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:24.581 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.853615363Z", 2020-11-19T03:06:24.582 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.582 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.582 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.582 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.583 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.583 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:24.583 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.584 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.584 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.584 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.585 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.585 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.857577_262.1", 2020-11-19T03:06:24.586 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.586 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:24.586 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.857577608Z", 2020-11-19T03:06:24.586 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.587 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.587 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.587 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.588 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.588 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:24.588 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.588 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.589 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.590 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.859745_263.1", 2020-11-19T03:06:24.590 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.590 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19", 2020-11-19T03:06:24.590 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.859745204Z", 2020-11-19T03:06:24.591 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.591 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.591 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.591 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.592 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.592 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Qy4UpA_MiVj8fLm1yoUPwmx", 2020-11-19T03:06:24.593 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.593 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.593 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.593 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.594 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.594 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.863407_264.1", 2020-11-19T03:06:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19", 2020-11-19T03:06:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.863407887Z", 2020-11-19T03:06:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Qy4UpA_MiVj8fLm1yoUPwmx", 2020-11-19T03:06:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.598 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.599 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.869018_265.1", 2020-11-19T03:06:24.599 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.599 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:24.602 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.869018103Z", 2020-11-19T03:06:24.602 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.603 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.603 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:24.603 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.604 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.604 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:24.604 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.605 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.605 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.606 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754783.874924_266.1", 2020-11-19T03:06:24.606 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.607 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19", 2020-11-19T03:06:24.607 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:43.874924462Z", 2020-11-19T03:06:24.607 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.607 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.607 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:24.608 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.608 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.608 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MtUHVBncx3w43A75VcchYpc", 2020-11-19T03:06:24.609 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.609 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.609 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.609 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.611 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754893.333355_282.1", 2020-11-19T03:06:24.611 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.611 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:24.611 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.333355116Z", 2020-11-19T03:06:24.612 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.612 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.612 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.612 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.613 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.613 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.613 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.614 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.614 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.615 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754893.338774_293.1", 2020-11-19T03:06:24.616 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.616 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:24.616 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.338774111Z", 2020-11-19T03:06:24.616 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.617 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.617 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.617 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.618 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.618 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.618 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.619 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.619 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.619 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.619 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.620 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754893.341581_294.1", 2020-11-19T03:06:24.620 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.621 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20", 2020-11-19T03:06:24.621 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.341581490Z", 2020-11-19T03:06:24.621 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.621 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.622 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.622 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.622 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.623 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_m4q0aNW1r6ONbHC6SDkHRNx", 2020-11-19T03:06:24.623 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.623 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.623 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.625 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754893.346605_295.1", 2020-11-19T03:06:24.625 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.625 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20", 2020-11-19T03:06:24.625 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:33.346605721Z", 2020-11-19T03:06:24.626 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.626 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.626 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.626 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.627 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.627 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_m4q0aNW1r6ONbHC6SDkHRNx", 2020-11-19T03:06:24.628 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.628 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.628 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.629 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754898.951630_299.1", 2020-11-19T03:06:24.630 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.630 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:24.630 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:38.951630338Z", 2020-11-19T03:06:24.631 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.631 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.631 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.631 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.632 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.632 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.632 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.633 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.633 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.634 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754898.962533_304.1", 2020-11-19T03:06:24.634 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.635 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:24.635 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:01:38.962533901Z", 2020-11-19T03:06:24.635 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.635 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.636 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.636 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.636 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.636 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.637 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.637 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.637 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.638 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754969.539612_314.1", 2020-11-19T03:06:24.639 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.639 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:24.639 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:02:49.539612219Z", 2020-11-19T03:06:24.639 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.640 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.640 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.640 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.641 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.641 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.641 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:24.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.642 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.642 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.643 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754969.549049_315.1", 2020-11-19T03:06:24.643 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.643 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20", 2020-11-19T03:06:24.644 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:02:49.549049417Z", 2020-11-19T03:06:24.644 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.644 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.644 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.645 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.645 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.645 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_l0hIIyTwYa8CDyvOWehDxvv", 2020-11-19T03:06:24.646 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:24.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.646 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.646 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.647 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.481687_339.1", 2020-11-19T03:06:24.648 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.648 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:24.648 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.481687459Z", 2020-11-19T03:06:24.648 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.649 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.649 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.649 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.650 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.650 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:24.650 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.651 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.651 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.652 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.486007_340.1", 2020-11-19T03:06:24.652 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.652 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:24.652 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.486007579Z", 2020-11-19T03:06:24.653 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.653 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.653 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.653 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.654 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.654 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:24.654 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.655 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.655 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.656 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.487864_341.1", 2020-11-19T03:06:24.656 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.657 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21", 2020-11-19T03:06:24.657 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.487864468Z", 2020-11-19T03:06:24.657 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.657 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.658 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.658 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.658 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.658 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UwTo_benAnYmaa4CegfHe2h", 2020-11-19T03:06:24.659 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.659 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.660 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.660 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.660 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.661 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.492471_342.1", 2020-11-19T03:06:24.661 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.661 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21", 2020-11-19T03:06:24.662 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.492471230Z", 2020-11-19T03:06:24.662 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.662 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.662 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.662 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.663 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.663 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UwTo_benAnYmaa4CegfHe2h", 2020-11-19T03:06:24.663 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.664 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.664 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.664 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.665 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.665 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.516426_343.1", 2020-11-19T03:06:24.665 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.666 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:24.666 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.516426936Z", 2020-11-19T03:06:24.666 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.667 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.667 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.667 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.668 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.668 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:24.668 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.669 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.669 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.669 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.669 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.670 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.520864_344.1", 2020-11-19T03:06:24.670 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.670 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:24.671 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.520864137Z", 2020-11-19T03:06:24.671 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.671 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.671 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.672 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.672 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.672 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:24.672 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.673 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.673 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.673 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.674 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.674 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.522824_345.1", 2020-11-19T03:06:24.674 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.675 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22", 2020-11-19T03:06:24.675 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.522824373Z", 2020-11-19T03:06:24.675 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.675 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.676 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.676 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.676 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.676 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-PCUgWQeHT2U7eFpqPZrnJx", 2020-11-19T03:06:24.677 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.677 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.677 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.678 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.527162_346.1", 2020-11-19T03:06:24.679 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.679 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22", 2020-11-19T03:06:24.679 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.527162988Z", 2020-11-19T03:06:24.679 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.680 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.680 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.680 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.680 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.681 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-PCUgWQeHT2U7eFpqPZrnJx", 2020-11-19T03:06:24.681 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.681 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.682 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.683 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.572422_347.1", 2020-11-19T03:06:24.683 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.683 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:24.683 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.572422879Z", 2020-11-19T03:06:24.683 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.684 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.684 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.684 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.685 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.685 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:24.685 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.686 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.686 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.687 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.578468_348.1", 2020-11-19T03:06:24.687 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.687 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:24.688 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.578468758Z", 2020-11-19T03:06:24.688 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.688 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.688 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.689 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.689 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.689 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:24.690 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.690 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.690 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.690 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.691 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.580555_349.1", 2020-11-19T03:06:24.692 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.692 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23", 2020-11-19T03:06:24.692 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.580555073Z", 2020-11-19T03:06:24.692 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.692 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.693 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.693 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.693 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.694 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_PtmtnLQARXBOkzZFBwIKFhM", 2020-11-19T03:06:24.694 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.694 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.694 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.695 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.585888_350.1", 2020-11-19T03:06:24.696 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.696 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23", 2020-11-19T03:06:24.696 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.585888316Z", 2020-11-19T03:06:24.696 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.697 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.697 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.697 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.697 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.698 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_PtmtnLQARXBOkzZFBwIKFhM", 2020-11-19T03:06:24.698 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.698 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.699 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.699 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.699 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.700 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.610407_351.1", 2020-11-19T03:06:24.700 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.700 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:24.700 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.610407033Z", 2020-11-19T03:06:24.700 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.701 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.701 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:24.701 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.702 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.702 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:24.702 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.702 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.703 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.704 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.616786_352.1", 2020-11-19T03:06:24.704 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.704 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21", 2020-11-19T03:06:24.704 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.616786647Z", 2020-11-19T03:06:24.705 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.705 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.705 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:24.705 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.706 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.706 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3yQ0HOny1I5G4CvqqSfF0VZ", 2020-11-19T03:06:24.707 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.707 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.707 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.708 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.708 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.625234_353.1", 2020-11-19T03:06:24.708 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.709 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:24.709 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.625234108Z", 2020-11-19T03:06:24.709 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.709 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.633175_354.1", 2020-11-19T03:06:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22", 2020-11-19T03:06:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.633175833Z", 2020-11-19T03:06:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2FSwJuzxbYIK9kKiX8f9Jip", 2020-11-19T03:06:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.644998_355.1", 2020-11-19T03:06:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.644998413Z", 2020-11-19T03:06:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755009.660851_356.1", 2020-11-19T03:06:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23", 2020-11-19T03:06:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:29.660851700Z", 2020-11-19T03:06:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-V-a_cNVKYEYfjY2qMUe0cW", 2020-11-19T03:06:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.005483_361.1", 2020-11-19T03:06:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.005483628Z", 2020-11-19T03:06:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.009877_374.1", 2020-11-19T03:06:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.009877496Z", 2020-11-19T03:06:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.012023_375.1", 2020-11-19T03:06:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24", 2020-11-19T03:06:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.012023407Z", 2020-11-19T03:06:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ycH_LWaCb7pN9H3I9w2Z6N_", 2020-11-19T03:06:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.016947_376.1", 2020-11-19T03:06:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24", 2020-11-19T03:06:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.016947542Z", 2020-11-19T03:06:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ycH_LWaCb7pN9H3I9w2Z6N_", 2020-11-19T03:06:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.045357_377.1", 2020-11-19T03:06:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.045357346Z", 2020-11-19T03:06:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.050883_378.1", 2020-11-19T03:06:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.050883011Z", 2020-11-19T03:06:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.053418_379.1", 2020-11-19T03:06:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25", 2020-11-19T03:06:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.053418708Z", 2020-11-19T03:06:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bkpajY1jnfqrwJDp6NVto29", 2020-11-19T03:06:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.059396_380.1", 2020-11-19T03:06:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25", 2020-11-19T03:06:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.059396024Z", 2020-11-19T03:06:24.756 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.756 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.756 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.756 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.757 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.757 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bkpajY1jnfqrwJDp6NVto29", 2020-11-19T03:06:24.757 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.092006_381.1", 2020-11-19T03:06:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.092006149Z", 2020-11-19T03:06:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.096908_382.1", 2020-11-19T03:06:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.096908426Z", 2020-11-19T03:06:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.099132_383.1", 2020-11-19T03:06:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26", 2020-11-19T03:06:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.099132448Z", 2020-11-19T03:06:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_WAoNjTvGKPea_y2zejmRi9k", 2020-11-19T03:06:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.109731_384.1", 2020-11-19T03:06:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26", 2020-11-19T03:06:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.109731627Z", 2020-11-19T03:06:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_WAoNjTvGKPea_y2zejmRi9k", 2020-11-19T03:06:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.128366_385.1", 2020-11-19T03:06:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.128366050Z", 2020-11-19T03:06:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.134273_386.1", 2020-11-19T03:06:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.134273445Z", 2020-11-19T03:06:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.138369_387.1", 2020-11-19T03:06:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.138369812Z", 2020-11-19T03:06:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.151920_388.1", 2020-11-19T03:06:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24", 2020-11-19T03:06:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.151920363Z", 2020-11-19T03:06:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ShqCx9Sbb33pLeWR11AIyaP", 2020-11-19T03:06:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.157598_389.1", 2020-11-19T03:06:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.157598827Z", 2020-11-19T03:06:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.167497_390.1", 2020-11-19T03:06:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.167497429Z", 2020-11-19T03:06:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.178487_391.1", 2020-11-19T03:06:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.178487831Z", 2020-11-19T03:06:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.186130_392.1", 2020-11-19T03:06:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25", 2020-11-19T03:06:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.186130781Z", 2020-11-19T03:06:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ITfbHxfEMl9gGadA_vt3sA3", 2020-11-19T03:06:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.200136_393.1", 2020-11-19T03:06:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.200136493Z", 2020-11-19T03:06:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.228723_394.1", 2020-11-19T03:06:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.228723559Z", 2020-11-19T03:06:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.241089_395.1", 2020-11-19T03:06:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.241089363Z", 2020-11-19T03:06:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755021.248167_396.1", 2020-11-19T03:06:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26", 2020-11-19T03:06:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:41.248167458Z", 2020-11-19T03:06:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b0qeecmkxaDwsMLZygJF4Xh", 2020-11-19T03:06:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2020-11-19T03:06:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.070651_403.1", 2020-11-19T03:06:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.070651958Z", 2020-11-19T03:06:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.075617_416.1", 2020-11-19T03:06:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.075617906Z", 2020-11-19T03:06:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.077978_417.1", 2020-11-19T03:06:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27", 2020-11-19T03:06:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.077978681Z", 2020-11-19T03:06:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jwhmoKKel5WXgN7QZadkAxM", 2020-11-19T03:06:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.081633_418.1", 2020-11-19T03:06:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27", 2020-11-19T03:06:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.081633830Z", 2020-11-19T03:06:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jwhmoKKel5WXgN7QZadkAxM", 2020-11-19T03:06:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.106122_419.1", 2020-11-19T03:06:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.106122712Z", 2020-11-19T03:06:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.845 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.845 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.845 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.846 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:24.846 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.846 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.846 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.847 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.847 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.848 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.111140_420.1", 2020-11-19T03:06:24.848 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.848 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:24.848 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.111140679Z", 2020-11-19T03:06:24.848 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.849 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.849 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.849 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.849 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.850 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:24.850 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.850 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.851 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.851 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.851 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.852 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.113526_421.1", 2020-11-19T03:06:24.852 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.852 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28", 2020-11-19T03:06:24.852 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.113526370Z", 2020-11-19T03:06:24.853 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.853 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.853 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.853 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.854 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.854 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_BALtCKJ9w_4jnJuGrJpKnG-", 2020-11-19T03:06:24.854 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.856 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.118961_422.1", 2020-11-19T03:06:24.856 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.856 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28", 2020-11-19T03:06:24.856 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.118961566Z", 2020-11-19T03:06:24.857 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.857 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.857 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.857 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.858 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.858 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_BALtCKJ9w_4jnJuGrJpKnG-", 2020-11-19T03:06:24.858 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.859 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.859 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.859 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.859 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.860 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.167348_423.1", 2020-11-19T03:06:24.860 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.860 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:24.861 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.167348445Z", 2020-11-19T03:06:24.861 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.861 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.861 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.862 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.862 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.862 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:24.862 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.863 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.863 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.864 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.864 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.172591_424.1", 2020-11-19T03:06:24.864 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.865 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:24.865 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.172591027Z", 2020-11-19T03:06:24.865 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.865 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.866 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.866 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.866 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.866 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:24.867 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.867 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.867 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.868 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.868 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.868 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.174666_425.1", 2020-11-19T03:06:24.869 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.869 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29", 2020-11-19T03:06:24.869 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.174666875Z", 2020-11-19T03:06:24.869 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.869 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.870 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.870 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.870 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.871 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7NmprpCcXUiFYkFKdzDwvBW", 2020-11-19T03:06:24.871 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.871 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.871 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.872 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.872 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.873 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.178503_426.1", 2020-11-19T03:06:24.873 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.873 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29", 2020-11-19T03:06:24.873 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.178503216Z", 2020-11-19T03:06:24.874 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.874 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.874 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.874 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.875 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.875 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7NmprpCcXUiFYkFKdzDwvBW", 2020-11-19T03:06:24.875 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.876 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.876 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.876 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.876 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.877 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.200545_427.1", 2020-11-19T03:06:24.877 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.877 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:24.878 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.200545045Z", 2020-11-19T03:06:24.878 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.878 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.878 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:24.878 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.879 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.879 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:24.879 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.880 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.880 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.881 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.208326_428.1", 2020-11-19T03:06:24.881 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.882 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-27:d71482d6-15fd-4715-9992-a25958dcb538.4341.27", 2020-11-19T03:06:24.882 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.208326306Z", 2020-11-19T03:06:24.882 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.882 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.882 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:24.883 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.883 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.883 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_K3c6CoFwjBhoNVpK0IvewKj", 2020-11-19T03:06:24.883 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.884 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.884 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.885 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.257261_429.1", 2020-11-19T03:06:24.885 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.886 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:24.886 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.257261094Z", 2020-11-19T03:06:24.886 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.886 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.886 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:24.887 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.887 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.887 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:24.888 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.888 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.888 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.889 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.271719_430.1", 2020-11-19T03:06:24.890 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.890 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-28:d71482d6-15fd-4715-9992-a25958dcb538.4341.28", 2020-11-19T03:06:24.890 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.271719934Z", 2020-11-19T03:06:24.890 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.890 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.891 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:24.891 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.891 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.891 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rWV4KAPTh1PSzVj7HEUHvTD", 2020-11-19T03:06:24.892 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.892 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.892 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.893 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.893 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.893 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.348435_431.1", 2020-11-19T03:06:24.894 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.894 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:24.894 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.348435126Z", 2020-11-19T03:06:24.894 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.894 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.895 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:24.895 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.895 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.896 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:24.896 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.896 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.896 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.898 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.358783_432.1", 2020-11-19T03:06:24.898 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.898 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-29:d71482d6-15fd-4715-9992-a25958dcb538.4341.29", 2020-11-19T03:06:24.898 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.358783093Z", 2020-11-19T03:06:24.898 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.899 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.899 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:24.899 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.900 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.900 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_dCis8Q8N3-GHeukiSg8ejA5", 2020-11-19T03:06:24.900 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.900 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.901 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.902 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.436658_433.1", 2020-11-19T03:06:24.902 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.902 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:24.902 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.436658111Z", 2020-11-19T03:06:24.903 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.903 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.903 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.903 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.904 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.904 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:24.904 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.905 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.905 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.905 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.905 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.906 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.443801_434.1", 2020-11-19T03:06:24.906 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.906 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:24.906 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.443801075Z", 2020-11-19T03:06:24.907 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.907 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.907 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.907 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.908 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.908 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:24.908 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.909 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.909 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.909 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.909 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.910 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.910 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.447377_435.1", 2020-11-19T03:06:24.910 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.910 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30", 2020-11-19T03:06:24.911 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.447377487Z", 2020-11-19T03:06:24.911 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.911 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.911 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.912 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.912 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.912 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_mM8-TW-E_epjEOhv1WllsAD", 2020-11-19T03:06:24.912 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.913 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.913 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.913 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.913 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.914 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.452011_436.1", 2020-11-19T03:06:24.914 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.915 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30", 2020-11-19T03:06:24.915 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.452011568Z", 2020-11-19T03:06:24.915 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.915 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.915 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.916 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.916 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.916 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_mM8-TW-E_epjEOhv1WllsAD", 2020-11-19T03:06:24.917 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.917 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.917 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.918 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.918 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.918 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.486369_437.1", 2020-11-19T03:06:24.919 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.919 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:24.919 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.486369972Z", 2020-11-19T03:06:24.919 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.919 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.920 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.920 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.920 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.920 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:24.921 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.921 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.921 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.921 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.922 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.922 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.922 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.490281_438.1", 2020-11-19T03:06:24.923 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.923 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:24.923 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.490281477Z", 2020-11-19T03:06:24.923 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.924 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.924 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.924 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.924 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.925 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:24.925 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.925 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.926 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.926 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.926 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.927 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.492125_439.1", 2020-11-19T03:06:24.927 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.928 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31", 2020-11-19T03:06:24.928 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.492125730Z", 2020-11-19T03:06:24.928 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.928 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.928 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.929 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.929 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.929 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gXleTzFvDf4A5fTuHAWH5iB", 2020-11-19T03:06:24.930 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.930 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.930 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.930 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.931 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.931 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.495843_440.1", 2020-11-19T03:06:24.932 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.932 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31", 2020-11-19T03:06:24.932 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.495843939Z", 2020-11-19T03:06:24.932 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.932 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.933 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.933 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.933 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.934 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gXleTzFvDf4A5fTuHAWH5iB", 2020-11-19T03:06:24.934 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.934 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.934 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.935 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.528063_441.1", 2020-11-19T03:06:24.936 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.936 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:24.936 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.528063552Z", 2020-11-19T03:06:24.936 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.937 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.937 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.937 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.937 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.938 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:24.938 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.938 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.939 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.940 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.532035_442.1", 2020-11-19T03:06:24.940 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.940 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:24.940 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.532035043Z", 2020-11-19T03:06:24.940 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.941 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.941 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.941 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.942 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.942 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:24.942 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.942 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.943 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.943 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.943 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.944 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.534111_443.1", 2020-11-19T03:06:24.944 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.944 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32", 2020-11-19T03:06:24.944 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.534111357Z", 2020-11-19T03:06:24.945 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.945 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.945 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.945 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.946 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.946 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_1MkPrP26FSf3qf6aGsjMwfM", 2020-11-19T03:06:24.946 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.947 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.947 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.948 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.544040_444.1", 2020-11-19T03:06:24.948 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.948 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32", 2020-11-19T03:06:24.949 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.544040853Z", 2020-11-19T03:06:24.949 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.949 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.949 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.950 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.950 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.950 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_1MkPrP26FSf3qf6aGsjMwfM", 2020-11-19T03:06:24.950 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.951 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.951 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.951 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.952 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.575866_445.1", 2020-11-19T03:06:24.953 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.953 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:24.953 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.575866003Z", 2020-11-19T03:06:24.953 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.953 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.954 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:24.954 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.954 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.955 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:24.955 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.955 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.957 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.585216_446.1", 2020-11-19T03:06:24.957 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.957 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-30:d71482d6-15fd-4715-9992-a25958dcb538.4341.30", 2020-11-19T03:06:24.957 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.585216241Z", 2020-11-19T03:06:24.957 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.958 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.958 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:24.958 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.959 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.959 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_e0h4uSYCicMRAxLyX3pShkT", 2020-11-19T03:06:24.959 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.959 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.960 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.960 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.960 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.961 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.639139_447.1", 2020-11-19T03:06:24.961 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.961 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:24.961 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.639139709Z", 2020-11-19T03:06:24.962 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.962 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.962 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:24.962 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.963 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.963 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:24.963 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.964 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.964 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.964 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.964 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.965 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.649165_448.1", 2020-11-19T03:06:24.965 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.965 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-31:d71482d6-15fd-4715-9992-a25958dcb538.4341.31", 2020-11-19T03:06:24.966 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.649165203Z", 2020-11-19T03:06:24.966 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.966 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.966 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:24.967 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.967 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.967 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UtR3A5JE3MhIEkJHXZ5L1mv", 2020-11-19T03:06:24.968 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.968 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.968 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.968 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.969 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.710088_449.1", 2020-11-19T03:06:24.970 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.970 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:24.970 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.710088485Z", 2020-11-19T03:06:24.970 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.971 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.971 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:24.971 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.972 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.972 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:24.972 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.972 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.973 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.973 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.973 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.974 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.720549_450.1", 2020-11-19T03:06:24.974 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.974 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-32:d71482d6-15fd-4715-9992-a25958dcb538.4341.32", 2020-11-19T03:06:24.975 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.720549878Z", 2020-11-19T03:06:24.975 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.975 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.975 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:24.975 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.976 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.976 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_H1d56_QAdLnRka47cqQiEaU", 2020-11-19T03:06:24.976 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:24.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.977 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.977 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.978 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.978 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.794865_451.1", 2020-11-19T03:06:24.978 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.979 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:24.979 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.794865388Z", 2020-11-19T03:06:24.979 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.979 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.980 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.980 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.980 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.981 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:24.981 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.981 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.981 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.982 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.982 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.983 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.799032_452.1", 2020-11-19T03:06:24.983 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.983 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:24.983 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.799032992Z", 2020-11-19T03:06:24.983 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.984 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.984 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.984 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.985 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.985 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:24.985 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.985 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.986 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.987 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.801245_453.1", 2020-11-19T03:06:24.987 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.987 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33", 2020-11-19T03:06:24.988 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.801245355Z", 2020-11-19T03:06:24.988 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.988 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.988 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.988 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.989 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.989 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8lKVoHs2FBMY1x8WlQGeuNM", 2020-11-19T03:06:24.989 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.990 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.990 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.991 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.991 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.805695_454.1", 2020-11-19T03:06:24.991 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:24.992 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33", 2020-11-19T03:06:24.992 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.805695695Z", 2020-11-19T03:06:24.992 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.992 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.993 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.993 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.993 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.994 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8lKVoHs2FBMY1x8WlQGeuNM", 2020-11-19T03:06:24.994 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.994 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.994 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:24.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:24.996 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.832169_455.1", 2020-11-19T03:06:24.996 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:24.996 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:24.996 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.832169972Z", 2020-11-19T03:06:24.996 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:24.997 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:24.997 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:24.997 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:24.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.998 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:24.998 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:24.998 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:24.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:24.999 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:24.999 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:24.999 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.999 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:24.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.000 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.837510_456.1", 2020-11-19T03:06:25.000 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.000 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:25.001 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.837510588Z", 2020-11-19T03:06:25.001 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.001 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.001 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.002 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.002 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.002 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:25.003 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.003 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.003 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.003 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.004 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.839242_457.1", 2020-11-19T03:06:25.005 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.005 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34", 2020-11-19T03:06:25.005 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.839242953Z", 2020-11-19T03:06:25.005 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.005 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.006 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.006 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.007 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.007 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uX_C_amZ6wHaMpUOaDbLhIh", 2020-11-19T03:06:25.007 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.007 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.008 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.008 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.008 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.009 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.843268_458.1", 2020-11-19T03:06:25.009 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.009 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34", 2020-11-19T03:06:25.010 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.843268836Z", 2020-11-19T03:06:25.010 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.010 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.010 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.011 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.011 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.011 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uX_C_amZ6wHaMpUOaDbLhIh", 2020-11-19T03:06:25.012 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.012 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.012 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.013 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.918282_459.1", 2020-11-19T03:06:25.014 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.014 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:25.014 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.918282306Z", 2020-11-19T03:06:25.014 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.015 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.015 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.015 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.015 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.016 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:25.016 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.016 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.017 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.017 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.017 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.018 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.927511_460.1", 2020-11-19T03:06:25.018 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.018 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:25.018 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.927511062Z", 2020-11-19T03:06:25.019 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.019 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.019 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.019 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.020 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.020 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:25.020 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.021 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.021 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.021 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.021 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.022 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.930242_461.1", 2020-11-19T03:06:25.022 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.022 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35", 2020-11-19T03:06:25.023 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.930242283Z", 2020-11-19T03:06:25.023 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.023 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.023 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.023 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.024 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.024 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LIlIyIKf54pQyVwNBj8qRry", 2020-11-19T03:06:25.024 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.025 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.025 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.025 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.025 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.026 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.935757_462.1", 2020-11-19T03:06:25.026 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.027 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35", 2020-11-19T03:06:25.027 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.935757929Z", 2020-11-19T03:06:25.027 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.027 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.028 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.028 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.028 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.028 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LIlIyIKf54pQyVwNBj8qRry", 2020-11-19T03:06:25.029 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.029 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.029 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.030 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.030 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.031 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.960222_463.1", 2020-11-19T03:06:25.031 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.031 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:25.031 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.960222969Z", 2020-11-19T03:06:25.032 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.032 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.032 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:25.032 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.033 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.033 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:25.033 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.034 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.034 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.034 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.034 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.035 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755034.966451_464.1", 2020-11-19T03:06:25.035 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.036 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-33:d71482d6-15fd-4715-9992-a25958dcb538.4341.33", 2020-11-19T03:06:25.036 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:54.966451383Z", 2020-11-19T03:06:25.036 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.036 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.036 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:25.037 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.037 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.037 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uVP1XWHqVMdf3mdyHJzx8nf", 2020-11-19T03:06:25.038 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.038 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.038 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.040 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755036.023079_465.1", 2020-11-19T03:06:25.040 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.040 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:25.040 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:56.023079326Z", 2020-11-19T03:06:25.040 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.041 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.041 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:25.041 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.042 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.042 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:25.042 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.043 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.043 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.043 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.043 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.044 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755036.036438_466.1", 2020-11-19T03:06:25.044 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.044 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-34:d71482d6-15fd-4715-9992-a25958dcb538.4341.34", 2020-11-19T03:06:25.045 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:56.036438127Z", 2020-11-19T03:06:25.045 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.045 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.045 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:25.046 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.046 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.046 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2d13u97DKl7AWpj5BetliLw", 2020-11-19T03:06:25.047 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.047 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.047 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.047 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.048 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.048 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.098379_467.1", 2020-11-19T03:06:25.049 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.049 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:25.049 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.098379164Z", 2020-11-19T03:06:25.049 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.050 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.050 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:25.050 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.051 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.051 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:25.051 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.051 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.052 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.052 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.052 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.053 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.106961_468.1", 2020-11-19T03:06:25.053 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.053 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-35:d71482d6-15fd-4715-9992-a25958dcb538.4341.35", 2020-11-19T03:06:25.054 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.106961657Z", 2020-11-19T03:06:25.054 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.054 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.054 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:25.055 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.055 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.055 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_7Pg4T88Tljn487S4R6hOTYY", 2020-11-19T03:06:25.056 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.056 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.056 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.057 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.057 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.057 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.187937_469.1", 2020-11-19T03:06:25.058 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.058 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36", 2020-11-19T03:06:25.058 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.187937642Z", 2020-11-19T03:06:25.058 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.059 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.059 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.059 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.060 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.060 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jRizeyH8HQBihpZqQ7H0cWx", 2020-11-19T03:06:25.060 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.061 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.061 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.062 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.191921_470.1", 2020-11-19T03:06:25.062 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.062 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36", 2020-11-19T03:06:25.063 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.191921477Z", 2020-11-19T03:06:25.063 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.063 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.063 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.064 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.064 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jRizeyH8HQBihpZqQ7H0cWx", 2020-11-19T03:06:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.066 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.066 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.193772_471.1", 2020-11-19T03:06:25.067 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.067 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-36", 2020-11-19T03:06:25.067 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.193772231Z", 2020-11-19T03:06:25.067 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.068 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.068 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.068 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kYTvwOB6WrelYhgfkJRvPMC", 2020-11-19T03:06:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.070 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.071 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.197598_472.1", 2020-11-19T03:06:25.071 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.071 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-36", 2020-11-19T03:06:25.072 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.197598901Z", 2020-11-19T03:06:25.072 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.072 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.072 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.072 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.073 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.073 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kYTvwOB6WrelYhgfkJRvPMC", 2020-11-19T03:06:25.073 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.074 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.074 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.075 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.235086_473.1", 2020-11-19T03:06:25.076 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.076 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37", 2020-11-19T03:06:25.076 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.235086410Z", 2020-11-19T03:06:25.076 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.076 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.077 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.077 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.077 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.077 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_my5SPqbG7dQQSEBcU42t2-N", 2020-11-19T03:06:25.078 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.078 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.078 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.079 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.079 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.080 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.239751_474.1", 2020-11-19T03:06:25.080 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.080 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37", 2020-11-19T03:06:25.080 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.239751394Z", 2020-11-19T03:06:25.080 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.081 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.081 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.081 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.082 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.082 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_my5SPqbG7dQQSEBcU42t2-N", 2020-11-19T03:06:25.082 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.083 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.083 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.083 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.083 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.084 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.242234_475.1", 2020-11-19T03:06:25.084 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.084 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-37", 2020-11-19T03:06:25.085 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.242234716Z", 2020-11-19T03:06:25.085 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.085 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.085 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.085 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.086 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.086 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_g91x9CrlSKmRRavm4MwY2r5", 2020-11-19T03:06:25.086 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.087 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.087 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.088 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.088 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755037.247176_476.1", 2020-11-19T03:06:25.089 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.089 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-37", 2020-11-19T03:06:25.089 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:57.247176520Z", 2020-11-19T03:06:25.089 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.089 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.090 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.090 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.090 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.091 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_g91x9CrlSKmRRavm4MwY2r5", 2020-11-19T03:06:25.091 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.091 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.092 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.092 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.092 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.093 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755038.595496_477.1", 2020-11-19T03:06:25.093 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.093 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38", 2020-11-19T03:06:25.093 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.595496690Z", 2020-11-19T03:06:25.094 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.094 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.094 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.094 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.095 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.095 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V7n32juggP5QmeLeOGxy3UK", 2020-11-19T03:06:25.095 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.096 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.096 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.096 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.096 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.097 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755038.599524_478.1", 2020-11-19T03:06:25.097 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.098 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38", 2020-11-19T03:06:25.098 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.599524203Z", 2020-11-19T03:06:25.098 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.098 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.098 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.099 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.099 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.099 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V7n32juggP5QmeLeOGxy3UK", 2020-11-19T03:06:25.100 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.100 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.100 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.102 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755038.601388_479.1", 2020-11-19T03:06:25.102 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.102 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-38", 2020-11-19T03:06:25.102 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.601388404Z", 2020-11-19T03:06:25.103 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.103 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.103 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.104 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.104 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_W1ZCfu17NUxLx7oS231exeH", 2020-11-19T03:06:25.104 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.105 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.105 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.106 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.106 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755038.605369_480.1", 2020-11-19T03:06:25.107 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.107 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-38", 2020-11-19T03:06:25.107 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:03:58.605369575Z", 2020-11-19T03:06:25.107 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.107 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.108 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.108 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.108 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.109 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_W1ZCfu17NUxLx7oS231exeH", 2020-11-19T03:06:25.109 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.109 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.109 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.110 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.110 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.111 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.318171_485.1", 2020-11-19T03:06:25.111 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.111 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-39:d71482d6-15fd-4715-9992-a25958dcb538.4341.39", 2020-11-19T03:06:25.111 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.318171704Z", 2020-11-19T03:06:25.112 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.112 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.112 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.112 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.113 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.113 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MR6DDMoW-bWzeQ2lVYjkv_T", 2020-11-19T03:06:25.113 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.114 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.114 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.115 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.326000_486.1", 2020-11-19T03:06:25.115 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.115 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-39:d71482d6-15fd-4715-9992-a25958dcb538.4341.39", 2020-11-19T03:06:25.116 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.326000603Z", 2020-11-19T03:06:25.116 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.116 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.116 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.117 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.117 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.117 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MR6DDMoW-bWzeQ2lVYjkv_T", 2020-11-19T03:06:25.117 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.118 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.118 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.118 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.119 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.119 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.328429_487.1", 2020-11-19T03:06:25.120 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.120 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-39", 2020-11-19T03:06:25.120 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.328429735Z", 2020-11-19T03:06:25.120 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.120 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.121 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.121 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.121 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.122 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2xs16O_Y3iMqtaOm0GzhjDK", 2020-11-19T03:06:25.122 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.122 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.122 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.123 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.123 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.124 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.333402_488.1", 2020-11-19T03:06:25.124 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.124 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-39", 2020-11-19T03:06:25.124 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.333402182Z", 2020-11-19T03:06:25.124 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.125 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.125 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.125 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.125 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.126 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2xs16O_Y3iMqtaOm0GzhjDK", 2020-11-19T03:06:25.126 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.126 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.127 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.127 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.127 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.128 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.366052_489.1", 2020-11-19T03:06:25.128 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.128 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-40:d71482d6-15fd-4715-9992-a25958dcb538.4341.40", 2020-11-19T03:06:25.128 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.366052352Z", 2020-11-19T03:06:25.129 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.129 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.129 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.129 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.130 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.130 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_oGE_O83RX1iSJnFe2p4H0J-", 2020-11-19T03:06:25.130 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.131 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.131 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.131 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.132 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.132 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.372774_490.1", 2020-11-19T03:06:25.133 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.133 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-40:d71482d6-15fd-4715-9992-a25958dcb538.4341.40", 2020-11-19T03:06:25.133 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.372774679Z", 2020-11-19T03:06:25.133 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.134 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.134 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.134 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.134 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.135 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_oGE_O83RX1iSJnFe2p4H0J-", 2020-11-19T03:06:25.135 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.135 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.136 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.136 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.136 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.137 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.375737_491.1", 2020-11-19T03:06:25.137 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.137 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-40", 2020-11-19T03:06:25.138 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.375737830Z", 2020-11-19T03:06:25.138 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.138 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.138 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.138 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.139 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.139 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bGn8OsdsLasWP-8EeCEuz7n", 2020-11-19T03:06:25.139 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.140 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.140 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.140 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.141 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.141 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.382371_492.1", 2020-11-19T03:06:25.142 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.142 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-40", 2020-11-19T03:06:25.142 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.382371633Z", 2020-11-19T03:06:25.142 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.142 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.143 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.143 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.143 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.144 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bGn8OsdsLasWP-8EeCEuz7n", 2020-11-19T03:06:25.144 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.144 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.145 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.145 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.145 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.146 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.429548_493.1", 2020-11-19T03:06:25.146 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.146 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-41:d71482d6-15fd-4715-9992-a25958dcb538.4341.41", 2020-11-19T03:06:25.146 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.429548543Z", 2020-11-19T03:06:25.147 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.147 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.147 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.147 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.148 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.148 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__wSJMtQIou6xYhPPtSS8YuA", 2020-11-19T03:06:25.148 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.149 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.149 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.149 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.149 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.150 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.436987_494.1", 2020-11-19T03:06:25.150 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.151 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-41:d71482d6-15fd-4715-9992-a25958dcb538.4341.41", 2020-11-19T03:06:25.151 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.436987958Z", 2020-11-19T03:06:25.151 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.151 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.152 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.152 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.152 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.153 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__wSJMtQIou6xYhPPtSS8YuA", 2020-11-19T03:06:25.153 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.153 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.154 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.155 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.439804_495.1", 2020-11-19T03:06:25.155 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.155 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-41", 2020-11-19T03:06:25.155 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.439804366Z", 2020-11-19T03:06:25.156 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.156 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.156 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.156 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.157 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.157 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_RsdLOBWehIFSw4jrAQphm1V", 2020-11-19T03:06:25.157 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.158 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.158 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.158 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.158 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.159 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755040.443579_496.1", 2020-11-19T03:06:25.159 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.160 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-41", 2020-11-19T03:06:25.160 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.443579496Z", 2020-11-19T03:06:25.160 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.160 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.160 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.161 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.161 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.163 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_RsdLOBWehIFSw4jrAQphm1V", 2020-11-19T03:06:25.163 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.164 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.164 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.164 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.164 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.165 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.049424_71.1", 2020-11-19T03:06:25.165 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.165 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-1:d71482d6-15fd-4715-9992-a25958dcb538.4341.1", 2020-11-19T03:06:25.166 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.049424391Z", 2020-11-19T03:06:25.166 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.166 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.166 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.167 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.167 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.167 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qgItci47XzVPPEUbRm9FP71", 2020-11-19T03:06:25.167 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.168 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.168 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.168 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.169 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.067007_75.1", 2020-11-19T03:06:25.169 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.170 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-1:d71482d6-15fd-4715-9992-a25958dcb538.4341.1", 2020-11-19T03:06:25.170 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.067007384Z", 2020-11-19T03:06:25.170 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.170 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.171 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.171 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.171 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.172 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qgItci47XzVPPEUbRm9FP71", 2020-11-19T03:06:25.172 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.172 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.173 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.174 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.068631_76.1", 2020-11-19T03:06:25.174 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.174 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-1", 2020-11-19T03:06:25.174 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.068631964Z", 2020-11-19T03:06:25.175 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.175 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.175 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.175 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.176 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.176 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_94mQ_WxhbwDk-kFSjKqwmYF", 2020-11-19T03:06:25.176 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.177 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.177 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.178 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.071728_77.1", 2020-11-19T03:06:25.178 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.179 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-1", 2020-11-19T03:06:25.179 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.071728952Z", 2020-11-19T03:06:25.179 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.179 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.180 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.180 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.180 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.181 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_94mQ_WxhbwDk-kFSjKqwmYF", 2020-11-19T03:06:25.181 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.181 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.181 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.182 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.182 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.183 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.094013_78.1", 2020-11-19T03:06:25.183 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.183 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-2:d71482d6-15fd-4715-9992-a25958dcb538.4341.2", 2020-11-19T03:06:25.183 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.094013583Z", 2020-11-19T03:06:25.184 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.184 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.184 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.184 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.185 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.185 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bSYBD3JTM-OkT97a99BJ1cL", 2020-11-19T03:06:25.185 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.186 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.186 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.187 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.098511_79.1", 2020-11-19T03:06:25.187 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.188 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-2:d71482d6-15fd-4715-9992-a25958dcb538.4341.2", 2020-11-19T03:06:25.188 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.098511431Z", 2020-11-19T03:06:25.188 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.188 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.189 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.189 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.189 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.189 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bSYBD3JTM-OkT97a99BJ1cL", 2020-11-19T03:06:25.190 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.190 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.190 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.191 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.191 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.192 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.100851_80.1", 2020-11-19T03:06:25.192 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.192 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-2", 2020-11-19T03:06:25.192 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.100851568Z", 2020-11-19T03:06:25.193 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.193 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.193 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.193 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.194 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.194 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5UuoR0SlRK-wajXJjy8RaHm", 2020-11-19T03:06:25.194 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.195 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.195 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.195 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.195 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.196 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754535.105586_81.1", 2020-11-19T03:06:25.196 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.197 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-2", 2020-11-19T03:06:25.197 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:35.105586168Z", 2020-11-19T03:06:25.197 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.197 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.198 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.198 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.198 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.198 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5UuoR0SlRK-wajXJjy8RaHm", 2020-11-19T03:06:25.199 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.199 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.199 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.200 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.200 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.200 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754538.238498_82.1", 2020-11-19T03:06:25.201 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.201 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-3:d71482d6-15fd-4715-9992-a25958dcb538.4341.3", 2020-11-19T03:06:25.201 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.238498971Z", 2020-11-19T03:06:25.202 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.202 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.202 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.202 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.203 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.203 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_GAW34u3sdJwjuChdK6wHKmm", 2020-11-19T03:06:25.203 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.204 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.204 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.204 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.205 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754538.263102_83.1", 2020-11-19T03:06:25.206 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.206 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-3:d71482d6-15fd-4715-9992-a25958dcb538.4341.3", 2020-11-19T03:06:25.206 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.263102074Z", 2020-11-19T03:06:25.206 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.207 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.207 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.207 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.208 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.208 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_GAW34u3sdJwjuChdK6wHKmm", 2020-11-19T03:06:25.208 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.208 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.209 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.209 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.209 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.210 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754538.265151_84.1", 2020-11-19T03:06:25.210 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.210 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-3", 2020-11-19T03:06:25.211 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.265151277Z", 2020-11-19T03:06:25.211 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.211 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.211 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.211 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.212 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.212 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tL7nc-Jhsvjv2EEZRXT73OW", 2020-11-19T03:06:25.212 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.213 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.213 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.213 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.213 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.214 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.214 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754538.270230_85.1", 2020-11-19T03:06:25.214 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.215 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-3", 2020-11-19T03:06:25.215 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:38.270230968Z", 2020-11-19T03:06:25.215 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.215 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.215 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.216 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.216 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.216 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tL7nc-Jhsvjv2EEZRXT73OW", 2020-11-19T03:06:25.217 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.217 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.217 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.218 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.218 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.646238_86.1", 2020-11-19T03:06:25.219 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.219 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-4:d71482d6-15fd-4715-9992-a25958dcb538.4341.4", 2020-11-19T03:06:25.219 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.646238073Z", 2020-11-19T03:06:25.219 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.220 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.220 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.220 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.220 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.221 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kXyu9LGpqsO6w8QSZAhud3g", 2020-11-19T03:06:25.221 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.221 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.221 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.221 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.222 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.222 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.222 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.223 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.653311_88.1", 2020-11-19T03:06:25.223 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.223 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-4:d71482d6-15fd-4715-9992-a25958dcb538.4341.4", 2020-11-19T03:06:25.223 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.653311568Z", 2020-11-19T03:06:25.223 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.224 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.224 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.224 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.224 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.225 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kXyu9LGpqsO6w8QSZAhud3g", 2020-11-19T03:06:25.225 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.225 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.225 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.226 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.226 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.226 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.227 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.655601_89.1", 2020-11-19T03:06:25.227 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.227 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-4", 2020-11-19T03:06:25.227 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.655601491Z", 2020-11-19T03:06:25.228 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.228 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.228 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.228 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.229 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.229 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tMmjLsHDWqdVdqcqT0YXzke", 2020-11-19T03:06:25.229 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.229 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.229 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.230 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.230 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.230 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.231 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.231 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.661306_90.1", 2020-11-19T03:06:25.231 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.231 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-4", 2020-11-19T03:06:25.231 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.661306950Z", 2020-11-19T03:06:25.232 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.232 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.232 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.232 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.233 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.233 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tMmjLsHDWqdVdqcqT0YXzke", 2020-11-19T03:06:25.233 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.234 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.234 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.234 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.234 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.235 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.235 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.685988_91.1", 2020-11-19T03:06:25.235 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.235 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-5:d71482d6-15fd-4715-9992-a25958dcb538.4341.5", 2020-11-19T03:06:25.236 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.685988313Z", 2020-11-19T03:06:25.236 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.236 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.236 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.236 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.237 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.237 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.237 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qVWvmOqSOj4nJ_IyDQjqQEP", 2020-11-19T03:06:25.237 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.238 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.238 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.238 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.238 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.238 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.239 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.239 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.690669_92.1", 2020-11-19T03:06:25.239 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.240 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-5:d71482d6-15fd-4715-9992-a25958dcb538.4341.5", 2020-11-19T03:06:25.240 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.690669131Z", 2020-11-19T03:06:25.240 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.240 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.241 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.241 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.241 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.241 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.241 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qVWvmOqSOj4nJ_IyDQjqQEP", 2020-11-19T03:06:25.242 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.242 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.242 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.243 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.243 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.243 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.243 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.693071_93.1", 2020-11-19T03:06:25.244 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.244 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-5", 2020-11-19T03:06:25.244 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.693071890Z", 2020-11-19T03:06:25.244 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.244 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.245 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.245 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.245 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.246 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IRRwWfE5ELTX6B9PeD0Wss2", 2020-11-19T03:06:25.246 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.246 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.246 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.247 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.247 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.697214_94.1", 2020-11-19T03:06:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-5", 2020-11-19T03:06:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.697214643Z", 2020-11-19T03:06:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IRRwWfE5ELTX6B9PeD0Wss2", 2020-11-19T03:06:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.740644_95.1", 2020-11-19T03:06:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-6:d71482d6-15fd-4715-9992-a25958dcb538.4341.6", 2020-11-19T03:06:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.740644196Z", 2020-11-19T03:06:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qrn-62ONHSx8esDlsD49N2S", 2020-11-19T03:06:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.744532_96.1", 2020-11-19T03:06:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-6:d71482d6-15fd-4715-9992-a25958dcb538.4341.6", 2020-11-19T03:06:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.744532390Z", 2020-11-19T03:06:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qrn-62ONHSx8esDlsD49N2S", 2020-11-19T03:06:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.746440_97.1", 2020-11-19T03:06:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-6", 2020-11-19T03:06:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.746440476Z", 2020-11-19T03:06:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_wFarfv5IWniL3C-MOrXA7X0", 2020-11-19T03:06:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754539.750854_98.1", 2020-11-19T03:06:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-6", 2020-11-19T03:06:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:39.750854665Z", 2020-11-19T03:06:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_wFarfv5IWniL3C-MOrXA7X0", 2020-11-19T03:06:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.678670_99.1", 2020-11-19T03:06:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7:d71482d6-15fd-4715-9992-a25958dcb538.4341.7", 2020-11-19T03:06:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.678670320Z", 2020-11-19T03:06:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3L9q5SfSgUsZej9VYTkemdm", 2020-11-19T03:06:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.684139_102.1", 2020-11-19T03:06:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7:d71482d6-15fd-4715-9992-a25958dcb538.4341.7", 2020-11-19T03:06:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.684139704Z", 2020-11-19T03:06:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3L9q5SfSgUsZej9VYTkemdm", 2020-11-19T03:06:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.686054_103.1", 2020-11-19T03:06:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7", 2020-11-19T03:06:25.278 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.686054368Z", 2020-11-19T03:06:25.278 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.278 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.278 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.278 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.279 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.279 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:25.279 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.690172_104.1", 2020-11-19T03:06:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7", 2020-11-19T03:06:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.690172550Z", 2020-11-19T03:06:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.714743_105.1", 2020-11-19T03:06:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8:d71482d6-15fd-4715-9992-a25958dcb538.4341.8", 2020-11-19T03:06:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.714743025Z", 2020-11-19T03:06:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Y2tOicXm39NSETUiNVpNZLc", 2020-11-19T03:06:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.719653_106.1", 2020-11-19T03:06:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8:d71482d6-15fd-4715-9992-a25958dcb538.4341.8", 2020-11-19T03:06:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.719653546Z", 2020-11-19T03:06:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Y2tOicXm39NSETUiNVpNZLc", 2020-11-19T03:06:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.722344_107.1", 2020-11-19T03:06:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8", 2020-11-19T03:06:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.722344069Z", 2020-11-19T03:06:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.295 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.295 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.295 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.295 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:25.296 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.296 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.296 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.297 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.727853_108.1", 2020-11-19T03:06:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8", 2020-11-19T03:06:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.727853163Z", 2020-11-19T03:06:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.773887_109.1", 2020-11-19T03:06:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9:d71482d6-15fd-4715-9992-a25958dcb538.4341.9", 2020-11-19T03:06:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.773887989Z", 2020-11-19T03:06:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vGJjA-wwP6rD2bH8RFgbNPS", 2020-11-19T03:06:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.778690_110.1", 2020-11-19T03:06:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9:d71482d6-15fd-4715-9992-a25958dcb538.4341.9", 2020-11-19T03:06:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.778690126Z", 2020-11-19T03:06:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vGJjA-wwP6rD2bH8RFgbNPS", 2020-11-19T03:06:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.781507_111.1", 2020-11-19T03:06:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9", 2020-11-19T03:06:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.781507411Z", 2020-11-19T03:06:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754542.787262_112.1", 2020-11-19T03:06:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9", 2020-11-19T03:06:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:42.787262520Z", 2020-11-19T03:06:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.124200_113.1", 2020-11-19T03:06:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7", 2020-11-19T03:06:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.124200729Z", 2020-11-19T03:06:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2020-11-19T03:06:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.128855_115.1", 2020-11-19T03:06:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-7", 2020-11-19T03:06:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.128855513Z", 2020-11-19T03:06:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XeiN7rqBAWblh8WXAStuoW7", 2020-11-19T03:06:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.183608_116.1", 2020-11-19T03:06:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8", 2020-11-19T03:06:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.183608180Z", 2020-11-19T03:06:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2020-11-19T03:06:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.189052_117.1", 2020-11-19T03:06:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-8", 2020-11-19T03:06:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.189052423Z", 2020-11-19T03:06:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_V-FQ-dNyw37gBAMnZrBf_09", 2020-11-19T03:06:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.288556_118.1", 2020-11-19T03:06:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9", 2020-11-19T03:06:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.288556017Z", 2020-11-19T03:06:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2020-11-19T03:06:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605754544.294580_119.1", 2020-11-19T03:06:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-9", 2020-11-19T03:06:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580366Z", 2020-11-19T03:06:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_6o8sARrEpS_Q-qLl94JTn5N", 2020-11-19T03:06:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2020-11-19T03:06:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:25.343 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:06:25.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:25.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:25.347 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0e78ea22-1fd1-48d9-9be5-2a207091b0ca --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:25.390 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:25.390 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:25.615 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.420742_202.1", 2020-11-19T03:06:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-42:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1", 2020-11-19T03:06:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.420742946Z", 2020-11-19T03:06:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_IB1_EODbelfVaTHE_5oAWhS", 2020-11-19T03:06:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.425874_203.1", 2020-11-19T03:06:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.626 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-42:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1", 2020-11-19T03:06:25.626 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.425874499Z", 2020-11-19T03:06:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_IB1_EODbelfVaTHE_5oAWhS", 2020-11-19T03:06:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.427631_204.1", 2020-11-19T03:06:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-42", 2020-11-19T03:06:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.427631612Z", 2020-11-19T03:06:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:25.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.639 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:25.640 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5Nz6qVWJsAJaG1rxvJOGYQB", 2020-11-19T03:06:25.641 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:25.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.642 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:25.643 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:25.643 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.644 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:25.645 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.431267_205.1", 2020-11-19T03:06:25.645 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:25.646 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-42", 2020-11-19T03:06:25.646 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.431267086Z", 2020-11-19T03:06:25.647 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:25.647 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:25.647 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:25.648 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:25.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.649 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:25.649 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5Nz6qVWJsAJaG1rxvJOGYQB", 2020-11-19T03:06:25.649 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:25.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.650 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:25.651 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:25.651 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.652 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:25.653 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.484260_206.1", 2020-11-19T03:06:25.654 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.655 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-43:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2", 2020-11-19T03:06:25.655 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.484260811Z", 2020-11-19T03:06:25.656 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:25.656 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:25.657 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:25.657 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:25.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.658 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:25.659 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2NTRsK9ffsy1L-z0DZdFmno", 2020-11-19T03:06:25.660 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:25.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.661 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:25.661 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:25.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.663 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:25.665 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.488512_207.1", 2020-11-19T03:06:25.666 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:25.666 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-43:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2", 2020-11-19T03:06:25.667 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.488512251Z", 2020-11-19T03:06:25.667 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:25.668 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:25.668 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:25.668 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:25.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.669 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:25.670 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2NTRsK9ffsy1L-z0DZdFmno", 2020-11-19T03:06:25.670 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:25.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.671 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:25.672 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:25.672 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.672 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:25.674 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.490480_208.1", 2020-11-19T03:06:25.674 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:25.675 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-43", 2020-11-19T03:06:25.675 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.490480678Z", 2020-11-19T03:06:25.675 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:25.676 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:25.676 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:25.677 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:25.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.677 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:25.678 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jkjotSu9mOACxRtgW9yZ10c", 2020-11-19T03:06:25.678 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:25.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.679 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:25.679 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:25.680 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.680 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:25.682 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.494399_209.1", 2020-11-19T03:06:25.682 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:25.682 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-43", 2020-11-19T03:06:25.683 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.494399862Z", 2020-11-19T03:06:25.683 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:25.684 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:25.684 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:25.685 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:25.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.686 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:25.686 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jkjotSu9mOACxRtgW9yZ10c", 2020-11-19T03:06:25.687 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:25.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:25.689 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:25.692 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:25.693 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.694 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.694 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:25.695 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:25.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:25.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:25.699 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0e78ea22-1fd1-48d9-9be5-2a207091b0ca --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:25.740 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:25.741 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:25.998 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:06:25.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:25.999 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.420742_202.1", 2020-11-19T03:06:25.999 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.000 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-42:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1", 2020-11-19T03:06:26.001 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.420742946Z", 2020-11-19T03:06:26.001 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.001 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.002 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.002 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.003 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.003 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IB1_EODbelfVaTHE_5oAWhS", 2020-11-19T03:06:26.004 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.005 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.005 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:26.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.011 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.425874_203.1", 2020-11-19T03:06:26.011 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.011 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-42:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1", 2020-11-19T03:06:26.011 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.425874499Z", 2020-11-19T03:06:26.011 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.012 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.012 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.012 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.013 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.013 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IB1_EODbelfVaTHE_5oAWhS", 2020-11-19T03:06:26.013 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.013 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.014 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:26.014 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.014 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.015 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.427631_204.1", 2020-11-19T03:06:26.015 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:26.015 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-42", 2020-11-19T03:06:26.015 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.427631612Z", 2020-11-19T03:06:26.016 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.016 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.016 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.016 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.017 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.017 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5Nz6qVWJsAJaG1rxvJOGYQB", 2020-11-19T03:06:26.017 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.018 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.018 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:26.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.019 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.431267_205.1", 2020-11-19T03:06:26.019 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:26.019 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-42", 2020-11-19T03:06:26.019 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.431267086Z", 2020-11-19T03:06:26.020 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.020 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.020 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.020 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.021 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.021 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5Nz6qVWJsAJaG1rxvJOGYQB", 2020-11-19T03:06:26.021 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.021 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.022 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:26.022 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.022 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.023 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.484260_206.1", 2020-11-19T03:06:26.023 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.023 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-43:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2", 2020-11-19T03:06:26.023 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.484260811Z", 2020-11-19T03:06:26.024 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.024 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.024 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.024 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.025 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.025 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2NTRsK9ffsy1L-z0DZdFmno", 2020-11-19T03:06:26.025 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.026 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.026 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:26.026 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.026 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.027 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.488512_207.1", 2020-11-19T03:06:26.027 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.027 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-43:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2", 2020-11-19T03:06:26.027 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.488512251Z", 2020-11-19T03:06:26.028 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.028 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.028 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.028 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.029 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.029 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2NTRsK9ffsy1L-z0DZdFmno", 2020-11-19T03:06:26.029 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.030 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.030 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:26.030 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.030 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.031 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.490480_208.1", 2020-11-19T03:06:26.031 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:26.031 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-43", 2020-11-19T03:06:26.031 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.490480678Z", 2020-11-19T03:06:26.032 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.032 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.032 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.032 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.033 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.033 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jkjotSu9mOACxRtgW9yZ10c", 2020-11-19T03:06:26.033 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.034 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.034 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:26.034 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.034 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.035 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.494399_209.1", 2020-11-19T03:06:26.035 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:26.035 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-43", 2020-11-19T03:06:26.036 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.494399862Z", 2020-11-19T03:06:26.036 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.036 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.036 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.036 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.037 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.037 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jkjotSu9mOACxRtgW9yZ10c", 2020-11-19T03:06:26.037 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.038 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.039 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:26.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.040 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:06:26.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:26.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:26.042 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0e78ea22-1fd1-48d9-9be5-2a207091b0ca --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:26.304 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:26.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.304 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.420742_202.1", 2020-11-19T03:06:26.305 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.305 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-42:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1", 2020-11-19T03:06:26.305 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.420742946Z", 2020-11-19T03:06:26.305 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.305 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.306 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.306 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.306 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.306 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_IB1_EODbelfVaTHE_5oAWhS", 2020-11-19T03:06:26.307 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.307 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.307 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:26.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.309 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.425874_203.1", 2020-11-19T03:06:26.309 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.309 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-42:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1", 2020-11-19T03:06:26.309 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.425874499Z", 2020-11-19T03:06:26.310 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.310 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.310 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.310 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.311 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.311 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_IB1_EODbelfVaTHE_5oAWhS", 2020-11-19T03:06:26.311 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.311 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.312 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:26.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.313 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.427631_204.1", 2020-11-19T03:06:26.313 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:26.313 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-42", 2020-11-19T03:06:26.313 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.427631612Z", 2020-11-19T03:06:26.313 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.314 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.314 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.314 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.314 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.315 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5Nz6qVWJsAJaG1rxvJOGYQB", 2020-11-19T03:06:26.315 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.315 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.315 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:26.316 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.316 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.316 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.431267_205.1", 2020-11-19T03:06:26.317 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:26.317 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-42", 2020-11-19T03:06:26.317 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.431267086Z", 2020-11-19T03:06:26.317 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.317 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.318 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.318 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.318 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.318 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_5Nz6qVWJsAJaG1rxvJOGYQB", 2020-11-19T03:06:26.319 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.319 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.319 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:26.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.320 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.320 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.484260_206.1", 2020-11-19T03:06:26.320 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.321 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-43:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2", 2020-11-19T03:06:26.321 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.484260811Z", 2020-11-19T03:06:26.321 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.321 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.321 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.322 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.322 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.322 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2NTRsK9ffsy1L-z0DZdFmno", 2020-11-19T03:06:26.322 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.323 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.323 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:26.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.324 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.488512_207.1", 2020-11-19T03:06:26.324 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.325 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-43:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2", 2020-11-19T03:06:26.325 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.488512251Z", 2020-11-19T03:06:26.325 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.325 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.326 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.326 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.326 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.326 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_2NTRsK9ffsy1L-z0DZdFmno", 2020-11-19T03:06:26.327 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.327 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.327 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:26.327 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.328 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.490480_208.1", 2020-11-19T03:06:26.328 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:26.329 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-43", 2020-11-19T03:06:26.329 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.490480678Z", 2020-11-19T03:06:26.329 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.329 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.330 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.330 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.330 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.330 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jkjotSu9mOACxRtgW9yZ10c", 2020-11-19T03:06:26.331 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.331 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.331 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:26.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.332 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755108.494399_209.1", 2020-11-19T03:06:26.332 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:26.333 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-43", 2020-11-19T03:06:26.333 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:05:08.494399862Z", 2020-11-19T03:06:26.333 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.333 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.333 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.334 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.334 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.334 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_jkjotSu9mOACxRtgW9yZ10c", 2020-11-19T03:06:26.334 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.335 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.337 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:26.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.338 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.338 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:26.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:26.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:26.340 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0e78ea22-1fd1-48d9-9be5-2a207091b0ca --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:26.378 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:26.378 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:26.627 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:06:26.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.628 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.420742_202.1", 2020-11-19T03:06:26.628 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.628 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-42:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1", 2020-11-19T03:06:26.628 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.420742946Z", 2020-11-19T03:06:26.628 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.629 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.629 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.630 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.630 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.630 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IB1_EODbelfVaTHE_5oAWhS", 2020-11-19T03:06:26.630 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.631 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.631 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:26.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.632 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.632 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.425874_203.1", 2020-11-19T03:06:26.632 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.633 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-42:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1", 2020-11-19T03:06:26.633 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.425874499Z", 2020-11-19T03:06:26.633 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.633 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.633 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.634 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.634 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.634 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IB1_EODbelfVaTHE_5oAWhS", 2020-11-19T03:06:26.634 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.635 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.635 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:26.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.636 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.427631_204.1", 2020-11-19T03:06:26.636 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:26.636 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-42", 2020-11-19T03:06:26.637 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.427631612Z", 2020-11-19T03:06:26.637 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.637 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.637 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.637 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.638 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.638 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5Nz6qVWJsAJaG1rxvJOGYQB", 2020-11-19T03:06:26.638 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.639 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.639 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:26.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.640 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.431267_205.1", 2020-11-19T03:06:26.640 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:26.640 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-42", 2020-11-19T03:06:26.640 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.431267086Z", 2020-11-19T03:06:26.641 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.641 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.641 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.641 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.642 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.642 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_5Nz6qVWJsAJaG1rxvJOGYQB", 2020-11-19T03:06:26.642 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.643 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.643 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:26.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.644 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.484260_206.1", 2020-11-19T03:06:26.644 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.644 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-43:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2", 2020-11-19T03:06:26.645 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.484260811Z", 2020-11-19T03:06:26.645 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.645 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.645 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.645 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.646 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.646 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2NTRsK9ffsy1L-z0DZdFmno", 2020-11-19T03:06:26.646 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.647 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.647 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:26.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.648 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.488512_207.1", 2020-11-19T03:06:26.648 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.648 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-43:c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2", 2020-11-19T03:06:26.648 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.488512251Z", 2020-11-19T03:06:26.649 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.649 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.649 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.649 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.650 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.650 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_2NTRsK9ffsy1L-z0DZdFmno", 2020-11-19T03:06:26.650 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.650 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.651 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:26.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.652 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.490480_208.1", 2020-11-19T03:06:26.652 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:26.652 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-43", 2020-11-19T03:06:26.652 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.490480678Z", 2020-11-19T03:06:26.652 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.653 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.653 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.653 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.653 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.654 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jkjotSu9mOACxRtgW9yZ10c", 2020-11-19T03:06:26.654 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.654 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.654 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:26.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:26.655 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755108.494399_209.1", 2020-11-19T03:06:26.656 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:26.656 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-43", 2020-11-19T03:06:26.656 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:05:08.494399862Z", 2020-11-19T03:06:26.656 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:26.656 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:26.657 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:26.657 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:26.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.657 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:26.657 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_jkjotSu9mOACxRtgW9yZ10c", 2020-11-19T03:06:26.657 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:26.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:26.658 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:26.659 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:26.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:26.660 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:06:26.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:26.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 43cc2be5-65c1-4813-aee6-d404c9e85e90 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:26.705 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:26.705 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:26.951 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:26.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.951 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.790634_393.1", 2020-11-19T03:06:26.951 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.952 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-44:d71482d6-15fd-4715-9992-a25958dcb538.5149.1", 2020-11-19T03:06:26.952 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.790634888Z", 2020-11-19T03:06:26.952 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.952 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.952 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.953 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.953 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.953 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_hHDaRo1tOfc5QcHVGxkzqVN", 2020-11-19T03:06:26.954 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.954 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.954 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:26.954 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.955 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.955 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.795136_394.1", 2020-11-19T03:06:26.955 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.956 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-44:d71482d6-15fd-4715-9992-a25958dcb538.5149.1", 2020-11-19T03:06:26.956 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.795136034Z", 2020-11-19T03:06:26.956 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.956 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.956 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.957 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.957 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.957 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_hHDaRo1tOfc5QcHVGxkzqVN", 2020-11-19T03:06:26.957 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.958 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.958 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:26.958 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.958 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.959 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.797249_395.1", 2020-11-19T03:06:26.959 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:26.959 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-44", 2020-11-19T03:06:26.960 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.797249018Z", 2020-11-19T03:06:26.960 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.960 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.960 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.960 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.961 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.961 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__tlYE_OQakwdKHTRt0v1oqF", 2020-11-19T03:06:26.961 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.962 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.962 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:26.962 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.962 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.963 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.801529_396.1", 2020-11-19T03:06:26.963 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:26.963 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-44", 2020-11-19T03:06:26.964 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.801529315Z", 2020-11-19T03:06:26.964 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.964 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.964 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.964 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.965 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.965 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__tlYE_OQakwdKHTRt0v1oqF", 2020-11-19T03:06:26.965 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.966 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.966 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:26.966 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.966 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.967 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.844874_397.1", 2020-11-19T03:06:26.967 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.967 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-45:d71482d6-15fd-4715-9992-a25958dcb538.5149.2", 2020-11-19T03:06:26.967 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.844874128Z", 2020-11-19T03:06:26.968 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.968 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.968 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.968 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.968 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.969 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_hrUTZbqdfdRsgsMGo2-EzV-", 2020-11-19T03:06:26.969 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.969 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.969 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:26.970 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.970 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.970 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.851709_398.1", 2020-11-19T03:06:26.971 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:26.971 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-45:d71482d6-15fd-4715-9992-a25958dcb538.5149.2", 2020-11-19T03:06:26.971 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.851709281Z", 2020-11-19T03:06:26.971 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.971 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.972 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.972 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.972 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.972 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_hrUTZbqdfdRsgsMGo2-EzV-", 2020-11-19T03:06:26.973 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.973 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.973 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:26.973 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.974 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.974 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.853877_399.1", 2020-11-19T03:06:26.975 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:26.975 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-45", 2020-11-19T03:06:26.975 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.853877295Z", 2020-11-19T03:06:26.975 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.975 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.976 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.976 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.976 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.976 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_HNmtrQkE6qW6engB8nn1_Hd", 2020-11-19T03:06:26.977 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.977 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.977 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:26.978 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.978 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:26.979 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.859139_400.1", 2020-11-19T03:06:26.979 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:26.979 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-45", 2020-11-19T03:06:26.979 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.859139473Z", 2020-11-19T03:06:26.979 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:26.980 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:26.980 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:26.980 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:26.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.980 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:26.981 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_HNmtrQkE6qW6engB8nn1_Hd", 2020-11-19T03:06:26.981 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:26.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:26.981 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:26.985 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:26.985 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.985 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.985 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:26.986 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:26.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:26.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:26.988 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 43cc2be5-65c1-4813-aee6-d404c9e85e90 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:27.028 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:27.028 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:27.264 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:06:27.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:27.265 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.790634_393.1", 2020-11-19T03:06:27.267 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:27.268 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-44:d71482d6-15fd-4715-9992-a25958dcb538.5149.1", 2020-11-19T03:06:27.268 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.790634888Z", 2020-11-19T03:06:27.268 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:27.268 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:27.268 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:27.269 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:27.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.269 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:27.269 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hHDaRo1tOfc5QcHVGxkzqVN", 2020-11-19T03:06:27.270 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:27.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.270 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:27.270 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:27.270 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.271 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:27.271 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.795136_394.1", 2020-11-19T03:06:27.271 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:27.272 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-44:d71482d6-15fd-4715-9992-a25958dcb538.5149.1", 2020-11-19T03:06:27.272 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.795136034Z", 2020-11-19T03:06:27.272 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:27.272 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:27.272 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:27.273 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:27.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.273 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:27.273 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hHDaRo1tOfc5QcHVGxkzqVN", 2020-11-19T03:06:27.273 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:27.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.274 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:27.274 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:27.274 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.274 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:27.275 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.797249_395.1", 2020-11-19T03:06:27.275 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:27.275 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-44", 2020-11-19T03:06:27.276 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.797249018Z", 2020-11-19T03:06:27.276 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:27.276 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:27.276 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:27.277 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:27.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.277 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:27.277 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__tlYE_OQakwdKHTRt0v1oqF", 2020-11-19T03:06:27.277 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:27.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.278 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:27.278 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:27.278 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.278 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:27.279 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.801529_396.1", 2020-11-19T03:06:27.279 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:27.279 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-44", 2020-11-19T03:06:27.280 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.801529315Z", 2020-11-19T03:06:27.280 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:27.280 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:27.280 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:27.280 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:27.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.281 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:27.281 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__tlYE_OQakwdKHTRt0v1oqF", 2020-11-19T03:06:27.281 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:27.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.282 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:27.282 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:27.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:27.284 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.844874_397.1", 2020-11-19T03:06:27.284 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:27.285 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-45:d71482d6-15fd-4715-9992-a25958dcb538.5149.2", 2020-11-19T03:06:27.285 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.844874128Z", 2020-11-19T03:06:27.285 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:27.285 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:27.286 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:27.286 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:27.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.286 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:27.286 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hrUTZbqdfdRsgsMGo2-EzV-", 2020-11-19T03:06:27.287 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:27.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.287 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:27.287 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:27.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:27.288 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.851709_398.1", 2020-11-19T03:06:27.288 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:27.289 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-45:d71482d6-15fd-4715-9992-a25958dcb538.5149.2", 2020-11-19T03:06:27.289 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.851709281Z", 2020-11-19T03:06:27.289 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:27.289 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:27.289 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:27.290 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:27.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.290 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:27.290 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hrUTZbqdfdRsgsMGo2-EzV-", 2020-11-19T03:06:27.290 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:27.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.291 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:27.291 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:27.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:27.292 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.853877_399.1", 2020-11-19T03:06:27.292 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:27.292 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-45", 2020-11-19T03:06:27.293 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.853877295Z", 2020-11-19T03:06:27.293 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:27.293 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:27.293 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:27.293 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:27.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.294 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:27.294 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_HNmtrQkE6qW6engB8nn1_Hd", 2020-11-19T03:06:27.294 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:27.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.295 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:27.295 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:27.295 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.295 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:27.296 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.859139_400.1", 2020-11-19T03:06:27.296 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:27.296 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-45", 2020-11-19T03:06:27.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.859139473Z", 2020-11-19T03:06:27.297 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:27.297 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:27.297 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:27.297 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:27.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.298 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:27.298 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_HNmtrQkE6qW6engB8nn1_Hd", 2020-11-19T03:06:27.298 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:27.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:27.298 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:27.303 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:27.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:27.304 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:06:27.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:27.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:27.307 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 43cc2be5-65c1-4813-aee6-d404c9e85e90 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:27.594 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:27.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:27.595 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.790634_393.1", 2020-11-19T03:06:27.595 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:27.595 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-44:d71482d6-15fd-4715-9992-a25958dcb538.5149.1", 2020-11-19T03:06:27.596 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.790634888Z", 2020-11-19T03:06:27.596 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:27.596 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:27.596 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:27.597 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:27.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.597 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:27.597 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_hHDaRo1tOfc5QcHVGxkzqVN", 2020-11-19T03:06:27.597 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:27.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.598 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:27.598 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:27.598 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:27.598 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:27.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:27.599 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.795136_394.1", 2020-11-19T03:06:27.599 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:27.599 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-44:d71482d6-15fd-4715-9992-a25958dcb538.5149.1", 2020-11-19T03:06:27.600 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.795136034Z", 2020-11-19T03:06:27.600 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:27.600 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:27.600 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:27.600 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:27.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.601 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:27.601 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_hHDaRo1tOfc5QcHVGxkzqVN", 2020-11-19T03:06:27.601 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:27.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.602 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:27.602 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:27.602 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:27.602 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:27.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:27.603 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.797249_395.1", 2020-11-19T03:06:27.603 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:27.603 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-44", 2020-11-19T03:06:27.604 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.797249018Z", 2020-11-19T03:06:27.604 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:27.604 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:27.604 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:27.604 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:27.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.605 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:27.605 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__tlYE_OQakwdKHTRt0v1oqF", 2020-11-19T03:06:27.605 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:27.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.606 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:27.606 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:27.606 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:27.606 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:27.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:27.607 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.801529_396.1", 2020-11-19T03:06:27.607 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:27.607 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-44", 2020-11-19T03:06:27.607 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.801529315Z", 2020-11-19T03:06:27.608 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:27.608 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:27.608 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:27.608 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:27.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:27.609 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:27.609 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "__tlYE_OQakwdKHTRt0v1oqF", 2020-11-19T03:06:28.583 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:28.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.584 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:28.584 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:28.584 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:28.585 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.844874_397.1", 2020-11-19T03:06:28.585 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:28.586 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-45:d71482d6-15fd-4715-9992-a25958dcb538.5149.2", 2020-11-19T03:06:28.586 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.844874128Z", 2020-11-19T03:06:28.586 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:28.586 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:28.587 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:28.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:28.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.587 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:28.587 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_hrUTZbqdfdRsgsMGo2-EzV-", 2020-11-19T03:06:28.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:28.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.588 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:28.588 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:28.588 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.589 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:28.589 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.851709_398.1", 2020-11-19T03:06:28.589 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket.instance", 2020-11-19T03:06:28.590 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-45:d71482d6-15fd-4715-9992-a25958dcb538.5149.2", 2020-11-19T03:06:28.590 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.851709281Z", 2020-11-19T03:06:28.590 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:28.590 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:28.590 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:28.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:28.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.591 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:28.591 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_hrUTZbqdfdRsgsMGo2-EzV-", 2020-11-19T03:06:28.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:28.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.592 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:28.592 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:28.592 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.592 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:28.593 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.853877_399.1", 2020-11-19T03:06:28.593 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:28.593 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-45", 2020-11-19T03:06:28.594 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.853877295Z", 2020-11-19T03:06:28.594 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:28.594 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:28.594 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:28.594 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:28.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.595 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:28.595 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_HNmtrQkE6qW6engB8nn1_Hd", 2020-11-19T03:06:28.595 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:28.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "write" 2020-11-19T03:06:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "1_1605755169.859139_400.1", 2020-11-19T03:06:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: "section": "bucket", 2020-11-19T03:06:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-45", 2020-11-19T03:06:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:09.859139473Z", 2020-11-19T03:06:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: "data": { 2020-11-19T03:06:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: "read_version": { 2020-11-19T03:06:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "", 2020-11-19T03:06:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 0 2020-11-19T03:06:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: "write_version": { 2020-11-19T03:06:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: "tag": "_HNmtrQkE6qW6engB8nn1_Hd", 2020-11-19T03:06:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": 1 2020-11-19T03:06:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: "status": { 2020-11-19T03:06:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "complete" 2020-11-19T03:06:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:28.602 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:28.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:28.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:28.603 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 43cc2be5-65c1-4813-aee6-d404c9e85e90 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:28.646 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:28.646 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:28.881 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:06:28.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:28.882 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.790634_393.1", 2020-11-19T03:06:28.882 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:28.882 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-44:d71482d6-15fd-4715-9992-a25958dcb538.5149.1", 2020-11-19T03:06:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.790634888Z", 2020-11-19T03:06:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hHDaRo1tOfc5QcHVGxkzqVN", 2020-11-19T03:06:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.885 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:28.885 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:28.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:28.886 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.795136_394.1", 2020-11-19T03:06:28.886 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:28.886 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-44:d71482d6-15fd-4715-9992-a25958dcb538.5149.1", 2020-11-19T03:06:28.887 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.795136034Z", 2020-11-19T03:06:28.887 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:28.887 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:28.887 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:28.887 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:28.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.888 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:28.888 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hHDaRo1tOfc5QcHVGxkzqVN", 2020-11-19T03:06:28.888 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:28.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.889 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:28.889 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:28.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.797249_395.1", 2020-11-19T03:06:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-44", 2020-11-19T03:06:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.797249018Z", 2020-11-19T03:06:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:28.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.892 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:28.892 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__tlYE_OQakwdKHTRt0v1oqF", 2020-11-19T03:06:28.892 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:28.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.893 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:28.893 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:28.893 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.893 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:28.894 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.801529_396.1", 2020-11-19T03:06:28.894 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:28.894 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-44", 2020-11-19T03:06:28.895 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.801529315Z", 2020-11-19T03:06:28.895 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:28.895 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:28.895 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:28.895 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:28.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.896 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:28.896 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "__tlYE_OQakwdKHTRt0v1oqF", 2020-11-19T03:06:28.896 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:28.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.897 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:28.897 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:28.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:28.898 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.844874_397.1", 2020-11-19T03:06:28.898 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:28.899 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-45:d71482d6-15fd-4715-9992-a25958dcb538.5149.2", 2020-11-19T03:06:28.899 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.844874128Z", 2020-11-19T03:06:28.899 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:28.899 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:28.900 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:28.900 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:28.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.900 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:28.900 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hrUTZbqdfdRsgsMGo2-EzV-", 2020-11-19T03:06:28.901 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:28.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.901 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:28.901 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:28.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.902 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.902 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:28.902 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.851709_398.1", 2020-11-19T03:06:28.903 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2020-11-19T03:06:28.903 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-45:d71482d6-15fd-4715-9992-a25958dcb538.5149.2", 2020-11-19T03:06:28.903 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.851709281Z", 2020-11-19T03:06:28.903 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:28.903 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:28.905 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:28.905 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:28.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.905 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:28.906 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hrUTZbqdfdRsgsMGo2-EzV-", 2020-11-19T03:06:28.906 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:28.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.906 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:28.906 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:28.907 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.907 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.907 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:28.907 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.853877_399.1", 2020-11-19T03:06:28.908 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:28.908 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-45", 2020-11-19T03:06:28.908 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.853877295Z", 2020-11-19T03:06:28.908 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:28.908 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:28.909 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:28.909 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:28.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.909 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:28.909 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_HNmtrQkE6qW6engB8nn1_Hd", 2020-11-19T03:06:28.910 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:28.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.910 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:28.910 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2020-11-19T03:06:28.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:28.911 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1605755169.859139_400.1", 2020-11-19T03:06:28.912 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2020-11-19T03:06:28.912 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-45", 2020-11-19T03:06:28.912 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:09.859139473Z", 2020-11-19T03:06:28.912 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2020-11-19T03:06:28.912 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2020-11-19T03:06:28.913 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2020-11-19T03:06:28.913 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2020-11-19T03:06:28.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.913 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2020-11-19T03:06:28.913 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_HNmtrQkE6qW6engB8nn1_Hd", 2020-11-19T03:06:28.914 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2020-11-19T03:06:28.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:28.914 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2020-11-19T03:06:28.915 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2020-11-19T03:06:28.915 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.916 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.916 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:28.916 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:06:28.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:28.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:28.917 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:06:28.955 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:28.955 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:29.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:29.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:29.251 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:06:29.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:29.287 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:29.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:29.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:29.756 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:06:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:30.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:30.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:30.060 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:06:30.094 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:30.095 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:30.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:30.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:30.499 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b2c8afed-318c-484c-8dd4-daab7a757391 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:30.808 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T03:06:30.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:30.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:30.811 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b2c8afed-318c-484c-8dd4-daab7a757391 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:30.853 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:30.853 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:31.132 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2020-11-19T03:06:31.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:31.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:31.137 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b2c8afed-318c-484c-8dd4-daab7a757391 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:31.441 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T03:06:31.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:31.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b2c8afed-318c-484c-8dd4-daab7a757391', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:31.449 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b2c8afed-318c-484c-8dd4-daab7a757391 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:31.491 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:31.492 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:31.753 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2020-11-19T03:06:31.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:31.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:31.758 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0e78ea22-1fd1-48d9-9be5-2a207091b0ca --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:31.799 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:31.800 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:32.058 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T03:06:32.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:32.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:32.063 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0e78ea22-1fd1-48d9-9be5-2a207091b0ca --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:32.104 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:32.104 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:32.358 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2020-11-19T03:06:32.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:32.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:32.361 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0e78ea22-1fd1-48d9-9be5-2a207091b0ca --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:32.402 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:32.402 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:32.649 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T03:06:33.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:33.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0e78ea22-1fd1-48d9-9be5-2a207091b0ca', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:33.749 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0e78ea22-1fd1-48d9-9be5-2a207091b0ca --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:33.791 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:33.792 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:34.057 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2020-11-19T03:06:34.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:34.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:34.060 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 43cc2be5-65c1-4813-aee6-d404c9e85e90 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:34.103 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:34.104 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:34.355 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T03:06:34.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:34.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:34.358 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 43cc2be5-65c1-4813-aee6-d404c9e85e90 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:34.398 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:34.399 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:34.668 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2020-11-19T03:06:34.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:34.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:34.671 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 43cc2be5-65c1-4813-aee6-d404c9e85e90 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:34.711 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:34.711 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:34.958 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T03:06:34.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:34.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '43cc2be5-65c1-4813-aee6-d404c9e85e90', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:34.960 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 43cc2be5-65c1-4813-aee6-d404c9e85e90 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:35.000 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:35.000 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:35.269 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2020-11-19T03:06:35.271 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-11-19T03:06:35.286 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-46 2020-11-19T03:06:35.332 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-47 2020-11-19T03:06:35.400 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-48 2020-11-19T03:06:35.511 INFO:tasks.rgw.c1.client.1.smithi086.stdout:2020-11-19T03:06:35.508+0000 7f97657ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:06:35.512 INFO:tasks.rgw.c2.client.1.smithi160.stdout:2020-11-19T03:06:35.505+0000 7f69b4ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:06:35.513 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2020-11-19T03:06:35.505+0000 7ff413fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:06:35.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:35.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:35.636 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:35.675 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:35.676 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:35.877 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:06:35.910 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:35.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:35.911 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:06:35.911 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:06:35.448366Z" 2020-11-19T03:06:35.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:35.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:35.911 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:35.912 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:06:35.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:35.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:35.912 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:35.913 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:06:35.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:35.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:35.913 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:35.913 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:06:35.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:35.914 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:35.915 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:06:35.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:35.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:35.916 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:35.952 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:35.952 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:36.172 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:06:36.172 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:06:36.173 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:06:36.173 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:06:36.173 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:06:36.173 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:06:36.173 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:36.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:36.174 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:06:36.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:36.174 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:06:36.174 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:36.175 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:36.175 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.437251_459.1", 2020-11-19T03:06:36.175 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:36.175 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:06:36.175 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:36.176 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.437251Z", 2020-11-19T03:06:36.176 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:36.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:36.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:36.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:36.177 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:06:36.177 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:36.177 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:36.177 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:06:36.178 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:36.178 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:36.178 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:36.178 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:06:36.178 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:36.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:36.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:36.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:36.179 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:06:36.179 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:36.180 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:36.180 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:36.180 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:36.180 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:36.180 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:36.181 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:36.181 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:36.181 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:36.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:36.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:36.182 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:06:36.182 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:36.182 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:36.182 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:36.183 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:36.183 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:36.183 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:36.183 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:36.183 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:36.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:36.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:36.184 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:06:36.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:36.184 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:06:36.185 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:06:36.185 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:06:36.185 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:36.185 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:06:36.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755195.448366_461.1 target=1_1605755195.437251_459.1 2020-11-19T03:06:41.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:41.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:41.189 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:41.227 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:41.228 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:41.486 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:06:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:06:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:06:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:06:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:06:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:06:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:06:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:06:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:06:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:06:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:06:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:06:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:06:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:06:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:06:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:06:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:41.497 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:41.497 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:41.497 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:41.497 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:06:41.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:41.498 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:06:41.498 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:06:41.498 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:06:41.498 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:41.498 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:06:41.499 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:06:41.500 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:06:41.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:41.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:41.501 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:41.542 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:41.542 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:41.799 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:06:41.799 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:06:41.799 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:06:41.799 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:06:41.800 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:06:41.800 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:06:41.800 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:06:41.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:41.800 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:06:41.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:41.801 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:06:41.801 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:41.801 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:41.802 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755195.437251_459.1", 2020-11-19T03:06:41.802 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:41.802 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:06:41.802 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:41.802 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:35.437251Z", 2020-11-19T03:06:41.803 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:06:41.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:41.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:41.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:41.803 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:06:41.804 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:41.804 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:41.804 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:06:41.804 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:41.804 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:41.805 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:41.805 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:06:41.805 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:41.805 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:41.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:41.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:41.806 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:06:41.806 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:41.806 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:41.806 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:41.807 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:41.807 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:41.807 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:41.807 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:06:41.807 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:41.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:41.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:41.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:41.808 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:06:41.808 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:41.809 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:41.809 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:41.809 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:41.809 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:41.809 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:41.810 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:06:41.810 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:41.810 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:41.810 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:41.810 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:06:41.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:41.811 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:06:41.811 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:06:41.811 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:06:41.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:41.812 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:06:41.813 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1605755195.448366_461.1 target=1_1605755195.437251_459.1 2020-11-19T03:06:46.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:46.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:46.816 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:46.866 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:47.102 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:06:47.102 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:06:47.103 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:06:47.103 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:06:47.103 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:06:47.103 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:06:47.104 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:06:47.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:47.104 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:06:47.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:47.104 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:06:47.105 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:47.105 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:47.105 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755195.437251_459.1", 2020-11-19T03:06:47.105 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:47.106 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:06:47.106 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:47.106 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:35.437251Z", 2020-11-19T03:06:47.106 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:06:47.106 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:47.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:47.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:47.107 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:06:47.107 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:47.107 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:47.108 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:06:47.108 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:47.108 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:47.108 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:47.108 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:06:47.109 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:47.109 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:47.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:47.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:47.110 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:06:47.110 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:47.110 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:47.111 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:47.111 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:47.111 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:47.111 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:47.111 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:06:47.112 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:47.112 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:47.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:47.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:47.113 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:06:47.113 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:47.113 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:47.113 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:47.113 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:47.114 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:47.114 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:47.114 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:06:47.114 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:47.114 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:47.115 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:47.115 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:06:47.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:47.115 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:06:47.115 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:06:47.116 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:06:47.116 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:47.116 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:06:47.117 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1605755195.448366_461.1 target=1_1605755195.437251_459.1 2020-11-19T03:06:52.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:52.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:52.494 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:52.534 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:52.535 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:52.785 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:06:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:06:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:06:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:06:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:06:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:06:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:06:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:06:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:06:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:06:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:06:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:06:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:06:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:06:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:06:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:06:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:06:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:06:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:06:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:06:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:06:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:06:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:06:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:06:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:06:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:06:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:06:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:52.799 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:06:52.800 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:06:52.801 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:06:52.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:52.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:52.802 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:52.846 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:52.847 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:53.122 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:06:53.122 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:06:53.122 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:06:53.122 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:06:53.123 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:06:53.123 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:06:53.123 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:53.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.124 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:06:53.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:53.124 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:06:53.124 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:53.124 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:53.125 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:06:53.125 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:53.125 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:06:53.125 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:53.125 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:06:53.126 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:06:53.126 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:53.126 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:06:53.127 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:53.127 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:53.127 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:06:53.127 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:53.127 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:53.128 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:53.128 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:06:53.128 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:53.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:53.129 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:06:53.129 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:53.129 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:53.129 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:53.130 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:53.130 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:53.130 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:53.130 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:53.130 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:53.131 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:53.131 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:06:53.131 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:53.132 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:06:53.132 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:06:53.132 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:53.132 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:53.132 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:53.133 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:06:53.133 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:06:53.133 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.133 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.134 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:06:53.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.135 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:06:53.135 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:06:53.135 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:06:53.135 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.136 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:06:53.137 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:06:53.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:53.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:53.138 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:53.185 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:53.186 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:53.466 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:53.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:53.467 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:06:53.467 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:06:53.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:53.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:53.467 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:53.468 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:06:53.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:53.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:53.468 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:53.468 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:06:53.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:53.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:53.469 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:53.469 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:06:53.469 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:53.470 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:53.470 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:06:53.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:53.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:53.472 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:53.515 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:53.515 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:53.781 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:06:53.781 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:06:53.781 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:06:53.782 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:06:53.782 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:06:53.782 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16121403893923359777 2020-11-19T03:06:53.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.782 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:06:53.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:53.783 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:06:53.783 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:53.783 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:53.784 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:06:53.784 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:53.784 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:53.784 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:53.784 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:06:53.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:53.785 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:06:53.785 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:53.786 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:53.786 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:53.786 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:53.786 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:53.786 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:53.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:06:53.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:53.787 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:06:53.788 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:53.788 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:53.788 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:53.788 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:53.788 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:53.789 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:53.789 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:06:53.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:53.790 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:06:53.791 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:53.791 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:53.791 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:53.791 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:53.792 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:53.792 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:53.792 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:06:53.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.793 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:06:53.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:53.793 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:06:53.793 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:06:53.793 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:06:53.795 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:53.795 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:06:53.796 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:06:53.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:53.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:53.797 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:53.841 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:53.842 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:54.114 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:54.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.114 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:06:54.114 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:06:54.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.115 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:54.115 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:06:54.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.116 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:54.116 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:06:54.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.117 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:54.117 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:06:54.117 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:54.117 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:54.118 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:06:54.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:54.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:54.120 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:54.200 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:54.201 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:54.464 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:06:54.464 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:06:54.465 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:06:54.465 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:06:54.465 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:06:54.465 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7000157538207389965 2020-11-19T03:06:54.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.466 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:06:54.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.466 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:06:54.466 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:54.467 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:54.467 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:06:54.467 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:54.468 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:54.468 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:54.468 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:06:54.469 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:54.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.469 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:06:54.470 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:54.470 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:54.470 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:54.470 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:54.471 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:54.471 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:54.471 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:06:54.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:54.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.472 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:06:54.473 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:54.473 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:54.473 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:54.473 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:54.474 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:54.474 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:54.474 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:06:54.474 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:54.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.475 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:06:54.475 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:54.476 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:54.476 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:54.476 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:54.476 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:54.476 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:54.477 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:06:54.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:54.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:54.477 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:06:54.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.478 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:06:54.478 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:06:54.478 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:06:54.479 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:54.479 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:06:54.480 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:06:54.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:54.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:54.482 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:54.565 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:54.565 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:54.832 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:54.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.833 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:06:54.833 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:06:54.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.833 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:54.834 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:06:54.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.834 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:54.834 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:06:54.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:54.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:54.835 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:54.835 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:06:54.836 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:54.836 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:54.837 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:06:54.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:54.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:54.838 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:54.885 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:54.885 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:55.155 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:06:55.155 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:06:55.156 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:06:55.156 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:06:55.156 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:06:55.156 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 12326036592617550149 2020-11-19T03:06:55.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:55.157 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:06:55.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:55.157 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:06:55.157 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:55.158 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:55.158 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:06:55.158 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:55.158 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:55.158 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:55.159 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:06:55.159 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:55.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:55.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:55.159 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:06:55.160 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:55.160 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:55.160 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:55.160 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:55.160 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:55.161 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:55.161 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:06:55.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:55.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:55.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:55.162 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:06:55.162 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:55.162 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:55.162 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:55.163 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:55.163 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:55.163 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:55.163 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:06:55.163 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:55.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:55.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:55.164 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:06:55.165 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:55.165 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:55.166 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:55.166 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:55.166 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:55.166 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:55.167 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:06:55.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:55.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:55.167 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:06:55.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:55.168 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:06:55.168 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:06:55.168 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:06:55.168 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:55.168 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:06:55.170 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:06:55.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:55.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:55.171 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:55.250 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:55.251 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:55.499 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:06:55.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:55.500 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:06:55.500 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:00:23.756845Z" 2020-11-19T03:06:55.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:55.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:55.501 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:55.501 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:04:00.372960Z" 2020-11-19T03:06:55.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:55.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:55.502 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:55.502 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T02:59:14.134469Z" 2020-11-19T03:06:55.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:55.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:55.502 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-19T03:06:55.503 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:06:35.779208Z" 2020-11-19T03:06:55.503 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:55.503 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:06:55.504 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:06:55.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:55.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:55.506 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:55.549 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:55.811 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:06:55.811 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:06:55.811 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:06:55.811 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:06:55.812 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:06:55.812 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7306459910885960034 2020-11-19T03:06:55.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:55.812 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:06:55.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:55.813 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:06:55.813 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:55.813 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:55.813 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:06:55.814 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:55.814 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:55.814 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:55.814 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:00:23.756845Z" 2020-11-19T03:06:55.814 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:55.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:55.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:55.815 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:06:55.815 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:55.815 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:55.816 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:55.816 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:55.816 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:55.816 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:55.816 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.372960Z" 2020-11-19T03:06:55.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:55.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:55.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:55.817 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:06:55.817 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:55.818 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:55.818 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:55.818 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:55.818 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:55.818 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:55.819 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:14.134469Z" 2020-11-19T03:06:55.819 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:55.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:55.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:55.819 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:06:55.820 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:55.820 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:55.820 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-19T03:06:55.820 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:55.820 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:55.821 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:55.821 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:35.779208Z" 2020-11-19T03:06:55.821 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:55.821 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:55.821 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:06:55.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:55.822 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:06:55.822 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:06:55.822 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:06:55.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:55.822 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:06:55.824 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:06:55.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:55.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:55.825 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:55.866 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:55.866 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:56.152 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:06:56.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:56.152 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:06:56.153 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:00:23.756845Z" 2020-11-19T03:06:56.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:56.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:56.154 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:56.154 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:04:00.372960Z" 2020-11-19T03:06:56.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:56.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:56.155 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:56.156 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T02:59:14.134469Z" 2020-11-19T03:06:56.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:56.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:56.157 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-19T03:06:56.157 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:06:35.779208Z" 2020-11-19T03:06:56.157 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:56.157 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:06:56.158 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:06:56.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:56.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:56.160 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:56.455 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:06:56.456 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:06:56.456 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:06:56.457 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:06:56.457 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:06:56.458 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 597164135494128594 2020-11-19T03:06:56.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:56.459 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:06:56.459 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:56.460 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:06:56.460 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:56.460 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:56.461 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:06:56.461 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:56.462 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:56.462 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:56.462 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:00:23.756845Z" 2020-11-19T03:06:56.463 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:56.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:56.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:56.464 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:06:56.464 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:56.464 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:56.465 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:56.465 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:56.465 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:56.466 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:56.466 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.372960Z" 2020-11-19T03:06:56.466 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:56.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:56.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:56.467 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:06:56.468 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:56.468 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:56.469 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:56.470 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:56.470 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:56.470 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:56.470 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:14.134469Z" 2020-11-19T03:06:56.470 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:56.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:56.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:56.471 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:06:56.471 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:56.471 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:56.472 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-19T03:06:56.472 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:56.472 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:56.472 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:56.472 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:35.779208Z" 2020-11-19T03:06:56.473 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:56.473 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:56.473 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:06:56.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:56.473 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:06:56.474 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:06:56.474 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:06:56.474 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:56.474 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:06:56.475 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:06:56.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:56.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:56.477 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:56.561 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:56.563 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:56.824 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:56.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:56.824 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:06:56.824 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.810439Z" 2020-11-19T03:06:56.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:56.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:56.825 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:56.825 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:04:00.336331Z" 2020-11-19T03:06:56.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:56.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:56.826 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:56.826 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.283402Z" 2020-11-19T03:06:56.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:56.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:56.827 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:56.827 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:06:35.733751Z" 2020-11-19T03:06:56.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:56.827 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:56.829 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:06:56.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:56.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:56.830 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:56.912 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:56.913 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:57.175 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:06:57.175 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:06:57.175 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:06:57.176 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:06:57.176 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:06:57.176 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 2448524765768921960 2020-11-19T03:06:57.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:57.176 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:06:57.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:57.177 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:06:57.177 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:57.177 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:57.178 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:06:57.178 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:57.178 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:57.178 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:57.178 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:00:23.810439Z" 2020-11-19T03:06:57.179 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:57.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:57.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:57.179 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:06:57.180 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:57.180 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:57.180 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:57.180 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:57.180 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:57.181 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:57.181 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:04:00.336331Z" 2020-11-19T03:06:57.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:57.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:57.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:57.182 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:06:57.182 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:57.182 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:57.182 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:57.183 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:57.183 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:57.183 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:57.183 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:59:14.283402Z" 2020-11-19T03:06:57.184 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:57.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:57.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:57.184 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:06:57.184 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:06:57.185 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:06:57.185 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:57.185 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:06:57.185 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:06:57.186 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:06:57.186 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:06:35.733751Z" 2020-11-19T03:06:57.186 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:57.187 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:57.187 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:06:57.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:57.187 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:06:57.188 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:06:57.188 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:06:57.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:57.189 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:06:57.190 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:06:57.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:57.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:57.192 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:57.293 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:57.860 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:57.861 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:57.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:57.862 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:06:57.863 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.810439Z" 2020-11-19T03:06:57.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:57.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:57.863 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:57.864 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:04:00.336331Z" 2020-11-19T03:06:57.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:57.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:57.864 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:57.865 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.283402Z" 2020-11-19T03:06:57.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:57.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:57.866 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:57.866 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:06:35.733751Z" 2020-11-19T03:06:57.866 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:57.867 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:57.867 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:06:57.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:57.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:57.869 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:57.917 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:57.918 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:58.178 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:06:58.178 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:06:58.179 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:06:58.179 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:06:58.179 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:06:58.179 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16483361435957904979 2020-11-19T03:06:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:06:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:06:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:58.181 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:58.181 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:06:58.181 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:58.181 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:58.182 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:58.182 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:00:23.810439Z" 2020-11-19T03:06:58.182 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:58.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:58.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:58.183 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:06:58.183 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:58.183 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:58.183 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:58.184 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:58.184 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:58.184 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:58.184 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.336331Z" 2020-11-19T03:06:58.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:58.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:58.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:58.185 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:06:58.185 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:58.186 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:58.186 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:58.186 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:58.186 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:58.186 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:58.187 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:14.283402Z" 2020-11-19T03:06:58.187 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:58.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:58.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:58.187 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:06:58.188 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:06:58.188 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:06:58.188 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:58.188 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:06:58.189 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:06:58.189 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:06:58.189 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.733751Z" 2020-11-19T03:06:58.189 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:58.190 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:58.190 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:06:58.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:58.190 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:06:58.190 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:06:58.191 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:06:58.191 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:58.191 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:06:58.192 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:06:58.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:58.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:58.194 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:58.228 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:58.229 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:58.482 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:58.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:58.483 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:06:58.483 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:06:58.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:58.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:58.484 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:58.484 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:06:58.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:58.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:58.485 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:58.485 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:06:58.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:58.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:58.486 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:58.486 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:06:58.486 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:58.486 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:58.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:58.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:58.489 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:06:58.569 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:58.569 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:58.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:58.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:58.846 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:58.894 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:58.895 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:59.148 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:06:59.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:59.149 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-11-19T03:06:59.149 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:00:23.756845Z" 2020-11-19T03:06:59.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:59.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:59.150 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:59.150 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:04:00.372960Z" 2020-11-19T03:06:59.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:59.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:59.154 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:59.154 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T02:59:14.134469Z" 2020-11-19T03:06:59.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:06:59.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:06:59.155 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-19T03:06:59.155 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:06:35.779208Z" 2020-11-19T03:06:59.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:06:59.156 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:06:59.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:59.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:59.158 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:06:59.237 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:59.237 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:59.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:59.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:06:59.523 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:06:59.561 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:59.561 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:06:59.836 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:06:59.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:59.837 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:06:59.837 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:00:23.681304Z" 2020-11-19T03:06:59.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:59.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:59.838 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:06:59.838 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:06:59.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:59.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:59.839 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:06:59.839 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:06:59.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:06:59.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:06:59.840 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:06:59.840 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:06:59.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:06:59.840 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:06:59.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:59.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:06:59.842 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:06:59.921 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:06:59.922 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:07:00.194 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-11-19T03:07:00.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:07:00.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:07:00.196 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:07:00.235 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:07:00.235 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:07:00.265 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:07:00.257+0000 7efc890bf0c0 0 failed reading obj info from .rgw.root:zone_info.d71482d6-15fd-4715-9992-a25958dcb538: (2) No such file or directory 2020-11-19T03:07:00.265 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:07:00.257+0000 7efc890bf0c0 0 WARNING: could not read zone params for zone id=d71482d6-15fd-4715-9992-a25958dcb538 name=test-zone1 2020-11-19T03:07:00.266 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:07:00.261+0000 7efc890bf0c0 0 failed reading obj info from .rgw.root:zone_info.fdf730f5-4390-45ac-9d84-6887e17cb661: (2) No such file or directory 2020-11-19T03:07:00.266 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:07:00.261+0000 7efc890bf0c0 0 WARNING: could not read zone params for zone id=fdf730f5-4390-45ac-9d84-6887e17cb661 name=test-zone3 2020-11-19T03:07:00.276 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:07:00.276 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:07:00.277 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:07:00.277 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-19T03:07:00.277 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-19T03:07:00.282 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-19T03:07:00.282 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-19T03:07:00.282 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-19T03:07:00.283 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-19T03:07:00.283 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-19T03:07:00.283 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-19T03:07:00.283 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-19T03:07:00.284 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-19T03:07:00.284 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-19T03:07:00.284 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-19T03:07:00.284 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-19T03:07:00.285 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-19T03:07:00.285 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T03:07:00.285 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T03:07:00.285 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T03:07:00.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.286 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T03:07:00.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.286 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T03:07:00.287 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:07:00.287 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-19T03:07:00.287 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T03:07:00.287 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T03:07:00.287 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-19T03:07:00.288 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2020-11-19T03:07:00.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:07:00.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.288 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-19T03:07:00.289 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T03:07:00.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:07:00.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:07:00.289 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.290 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:07:00.290 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-19T03:07:00.290 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:07:00.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:07:00.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:07:00.293 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:07:00.372 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:07:00.372 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:07:00.648 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone d71482d6-15fd-4715-9992-a25958dcb538 2020-11-19T03:07:00.737 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:07:00.738 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:07:00.738 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 2, 2020-11-19T03:07:00.738 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:07:00.738 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2020-11-19T03:07:00.739 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1605755108.494399_209.1", 2020-11-19T03:07:00.739 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:07:00.739 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:07:00.739 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2020-11-19T03:07:00.739 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.740 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2020-11-19T03:07:00.740 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:07:00.740 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2020-11-19T03:07:00.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.740 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2020-11-19T03:07:00.741 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2020-11-19T03:07:00.741 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2020-11-19T03:07:00.741 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": "true", 2020-11-19T03:07:00.741 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:07:00.741 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:07:00.742 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.742 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2020-11-19T03:07:00.742 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2020-11-19T03:07:00.742 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:07:00.742 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2020-11-19T03:07:00.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.743 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:07:00.743 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2020-11-19T03:07:00.743 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:07:00.744 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:07:00.744 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.744 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:07:00.744 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:07:00.744 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:07:00.744 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:07:00.745 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2020-11-19T03:07:00.745 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "false", 2020-11-19T03:07:00.745 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2020-11-19T03:07:00.746 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2020-11-19T03:07:00.746 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.746 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:07:00.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.747 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:07:00.747 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:07:00.748 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:07:00.748 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:07:00.748 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.748 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:07:00.748 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:07:00.749 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:07:00.749 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:07:00.749 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:07:00.749 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "false", 2020-11-19T03:07:00.751 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:07:00.751 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:07:00.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.752 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:07:00.752 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2020-11-19T03:07:00.752 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:07:00.753 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:07:00.753 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.753 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:07:00.753 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:07:00.753 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:07:00.754 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:07:00.754 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:07:00.754 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:07:00.754 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:07:00.754 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:07:00.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.755 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:07:00.755 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2020-11-19T03:07:00.755 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:07:00.756 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:07:00.756 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.756 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:07:00.756 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:07:00.756 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:07:00.757 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:07:00.757 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:07:00.757 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:07:00.757 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:07:00.757 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:07:00.758 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:07:00.758 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.758 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2020-11-19T03:07:00.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.758 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2020-11-19T03:07:00.759 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2020-11-19T03:07:00.759 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2020-11-19T03:07:00.759 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2020-11-19T03:07:00.759 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:07:00.759 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:07:00.760 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.760 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2020-11-19T03:07:00.760 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:07:00.761 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2020-11-19T03:07:00.761 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2020-11-19T03:07:00.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:07:00.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:07:00.761 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:07:00.765 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2020-11-19T03:07:00.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.766 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:07:00.766 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 1446762810 2020-11-19T03:07:00.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.767 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:07:00.767 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2049584274 2020-11-19T03:07:00.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.767 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:07:00.768 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2043733610 2020-11-19T03:07:00.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:07:00.768 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:07:00.768 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2362875045 2020-11-19T03:07:00.769 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:07:00.769 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:07:00.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.769 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:07:00.770 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:07:00.770 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2020-11-19T03:07:00.770 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2020-11-19T03:07:00.770 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:07:00.770 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:07:00.771 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:07:00.771 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:07:00.771 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:07:00.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.771 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2020-11-19T03:07:00.772 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:07:00.772 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:07:00.772 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:07:00.772 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:07:00.772 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:07:00.772 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:07:00.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:07:00.773 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:07:00.773 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2020-11-19T03:07:00.773 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:07:00.773 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:07:00.775 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-11-19T03:08:00.776 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-49 2020-11-19T03:08:00.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:00.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:00.891 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:00.931 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:00.932 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:01.146 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:08:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:08:01.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:08:01.179 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:08:01.179 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:08:00.842086Z" 2020-11-19T03:08:01.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:08:01.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:08:01.180 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:08:01.181 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:08:01.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:08:01.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:08:01.183 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:08:01.183 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:08:01.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:08:01.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:08:01.185 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:08:01.185 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:08:01.191 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:08:01.191 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:08:01.194 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:08:01.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:01.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:01.199 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:01.249 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:01.250 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:01.512 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:01.512 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:01.513 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:01.513 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:01.513 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:01.514 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:01.514 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:01.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:01.514 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:01.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:01.515 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:01.515 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:01.515 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:01.515 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:01.516 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:01.516 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:01.516 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:01.516 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:01.516 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:01.517 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:01.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:01.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:01.517 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:01.517 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:01.518 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:01.518 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:01.518 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:01.518 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:01.518 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:01.519 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:01.519 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:01.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:01.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:01.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:01.520 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:01.520 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:01.520 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:01.520 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:01.520 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:01.521 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:01.521 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:01.521 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:01.521 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:01.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:01.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:01.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:01.522 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:01.522 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:01.523 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:01.523 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:01.523 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:01.523 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:01.523 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:01.524 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:01.524 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:01.524 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:01.525 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:01.525 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:01.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:01.525 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:01.526 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:01.526 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:01.526 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:01.526 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:01.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:06.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:06.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:06.530 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:06.584 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:06.584 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:06.857 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:06.857 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:06.857 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:06.858 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:06.858 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:06.858 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:06.859 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:06.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:06.859 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:06.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:06.860 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:06.860 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:06.860 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:06.860 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:06.861 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:06.861 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:06.861 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:06.861 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:06.861 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:06.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:06.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:06.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:06.863 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:06.863 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:06.863 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:06.863 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:06.864 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:06.864 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:06.864 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:06.864 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:06.864 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:06.865 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:06.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:06.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:06.865 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:06.865 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:06.866 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:06.866 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:06.866 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:06.866 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:06.867 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:06.867 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:06.867 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:06.867 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:06.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:06.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:06.868 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:06.868 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:06.868 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:06.868 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:06.869 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:06.869 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:06.869 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:06.870 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:06.870 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:06.870 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:06.870 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:06.870 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:06.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:06.871 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:06.871 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:06.871 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:06.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:06.872 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:06.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:11.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:11.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:11.879 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:11.927 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:11.928 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:12.191 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:12.191 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:12.192 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:12.192 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:12.193 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:12.195 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:12.195 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:12.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:12.197 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:12.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:12.198 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:12.198 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:12.199 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:12.199 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:12.200 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:12.201 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:12.201 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:12.202 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:12.202 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:12.203 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:12.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:12.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:12.205 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:12.206 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:12.206 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:12.206 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:12.207 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:12.207 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:12.207 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:12.208 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:12.208 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:12.208 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:12.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:12.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:12.209 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:12.209 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:12.210 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:12.210 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:12.210 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:12.211 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:12.211 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:12.211 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:12.212 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:12.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:12.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:12.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:12.213 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:12.213 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:12.213 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:12.214 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:12.214 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:12.214 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:12.215 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:12.215 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:12.215 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:12.216 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:12.216 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:12.216 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:12.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:12.217 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:12.217 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:12.217 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:12.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:12.218 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:12.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:17.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:17.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:17.222 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:17.266 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:17.266 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:17.529 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:17.530 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:17.530 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:17.530 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:17.530 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:17.531 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:17.531 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:17.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:17.531 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:17.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:17.532 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:17.532 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:17.532 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:17.532 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:17.533 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:17.533 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:17.533 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:17.533 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:17.533 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:17.534 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:17.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:17.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:17.534 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:17.534 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:17.535 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:17.535 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:17.535 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:17.535 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:17.535 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:17.536 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:17.536 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:17.536 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:17.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:17.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:17.537 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:17.537 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:17.537 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:17.537 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:17.537 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:17.538 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:17.538 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:17.538 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:17.538 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:17.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:17.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:17.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:17.539 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:17.539 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:17.539 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:17.540 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:17.540 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:17.540 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:17.540 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:17.540 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:17.541 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:17.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:17.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:17.541 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:17.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:17.542 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:17.542 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:17.542 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:17.542 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:17.542 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:17.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:22.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:22.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:22.546 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:22.587 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:22.587 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:22.855 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:22.855 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:22.855 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:22.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:22.856 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:22.856 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:22.856 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:22.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:22.856 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:22.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:22.857 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:22.857 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:22.857 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:22.858 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:22.858 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:22.858 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:22.858 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:22.858 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:22.859 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:22.859 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:22.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:22.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:22.859 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:22.860 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:22.860 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:22.860 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:22.860 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:22.860 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:22.861 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:22.861 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:22.861 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:22.861 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:22.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:22.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:22.862 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:22.862 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:22.862 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:22.862 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:22.863 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:22.863 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:22.863 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:22.863 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:22.863 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:22.864 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:22.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:22.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:22.864 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:22.865 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:22.865 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:22.865 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:22.865 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:22.865 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:22.866 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:22.866 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:22.866 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:22.866 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:22.866 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:22.867 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:22.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:22.867 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:22.867 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:22.867 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:22.868 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:22.868 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:22.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:27.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:28.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:28.574 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:28.617 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:28.618 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:28.880 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:28.881 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:28.881 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:28.881 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:28.881 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:28.882 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:28.882 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:28.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:28.882 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:28.883 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:28.884 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:28.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:28.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:28.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:28.885 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:28.886 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:28.887 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:28.887 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:28.887 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:28.887 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:28.888 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:28.888 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:28.888 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:28.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:28.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:28.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:28.889 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:28.889 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:28.889 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:28.890 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:28.891 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:28.892 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:28.892 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:28.892 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:28.892 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:28.893 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:28.893 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:28.893 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:28.893 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:28.894 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:28.894 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:28.894 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:28.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:28.894 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:28.895 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:28.895 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:28.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:28.895 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:28.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:31.035 INFO:tasks.rgw.c1.client.1.smithi086.stdout:2020-11-19T03:08:31.029+0000 7f976cff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:08:33.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:33.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:33.898 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:33.951 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:33.951 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:34.206 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:34.206 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:34.206 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:34.206 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:34.207 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:34.207 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:34.207 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:34.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:34.208 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:34.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:34.208 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:34.208 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:34.208 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:34.209 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:34.209 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:34.209 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:34.209 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:34.209 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:34.210 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:34.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:34.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:34.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:34.210 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:34.211 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:34.211 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:34.211 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:34.211 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:34.211 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:34.212 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:34.212 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:34.212 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:34.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:34.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:34.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:34.213 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:34.213 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:34.213 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:34.214 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:34.214 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:34.214 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:34.214 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:34.215 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:34.215 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:34.215 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:34.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:34.216 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:34.216 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:34.216 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:34.216 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:34.216 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:34.217 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:34.217 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:34.217 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:34.217 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:34.217 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:34.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:34.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:34.218 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:34.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:34.218 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:34.219 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:34.219 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:34.219 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:34.220 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:34.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:39.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:39.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:39.225 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:39.269 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:39.269 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:39.557 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:44.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:44.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:44.565 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:44.604 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:44.604 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:44.883 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:44.883 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:44.883 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:44.884 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:44.884 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:44.884 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:44.884 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:44.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:44.885 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:44.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:44.885 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:44.886 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:44.886 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:44.886 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:44.886 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:44.886 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:44.887 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:44.887 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:44.887 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:44.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:44.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:44.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:44.888 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:44.888 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:44.888 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:44.889 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:44.889 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:44.889 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:44.889 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:44.889 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:44.890 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:44.890 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:44.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:44.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:44.891 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:44.891 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:44.891 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:44.891 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:44.891 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:44.892 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:44.892 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:44.892 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:44.892 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:44.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:44.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:44.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:44.893 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:44.893 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:44.894 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:44.894 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:44.894 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:44.894 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:44.894 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:44.895 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:44.895 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:44.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:44.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:44.896 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:44.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:44.896 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:44.896 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:44.897 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:44.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:44.897 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:44.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:49.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:49.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:49.901 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:49.943 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:49.943 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:50.205 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:50.206 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:50.206 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:50.206 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:50.206 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:50.207 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:50.207 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:50.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:50.207 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:50.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:50.208 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:50.208 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:50.208 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:50.208 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:50.209 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:50.209 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:50.209 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:50.209 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:50.209 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:50.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:50.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:50.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:50.210 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:50.210 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:50.211 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:50.211 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:50.211 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:50.211 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:50.211 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:50.212 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:50.212 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:50.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:50.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:50.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:50.213 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:50.213 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:50.213 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:50.213 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:50.213 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:50.218 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:50.218 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:50.218 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:50.219 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:50.219 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:50.219 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:50.219 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:50.219 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:50.220 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:50.220 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:50.220 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:50.220 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:50.220 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:50.221 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:50.221 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:50.221 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:50.221 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:50.221 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:50.222 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:50.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:50.222 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:50.222 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:50.222 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:50.223 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:50.223 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:50.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:08:55.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:55.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:08:55.230 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:08:55.272 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:08:55.273 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:08:55.535 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:08:55.535 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:08:55.535 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:08:55.536 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:08:55.536 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:08:55.536 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:08:55.536 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:55.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:55.538 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:08:55.538 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:55.538 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:08:55.538 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:55.539 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:08:55.539 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:08:55.539 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:55.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:55.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:55.540 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:08:55.540 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:55.540 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:55.540 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:08:55.540 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:55.541 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:55.541 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:55.541 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:08:55.541 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:55.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:55.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:55.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:55.542 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:08:55.542 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:55.542 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:55.543 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:55.543 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:55.543 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:55.543 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:55.544 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:55.544 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:55.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:55.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:55.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:08:55.545 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:08:55.545 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:08:55.545 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:08:55.545 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:08:55.546 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:08:55.546 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:08:55.546 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:08:55.546 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:08:55.546 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:08:55.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:55.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:55.547 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:08:55.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:08:55.547 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:08:55.548 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:08:55.548 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:08:55.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:08:55.548 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:08:55.549 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:09:00.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:09:00.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:09:00.552 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:09:00.606 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:09:00.606 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:09:00.899 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:09:00.899 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:09:00.899 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:09:00.900 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:09:00.900 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:09:00.900 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:09:00.900 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:09:00.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:00.901 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:09:00.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:00.901 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:09:00.901 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:00.902 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:00.902 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755195.448366_461.1", 2020-11-19T03:09:00.902 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:00.902 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:09:00.902 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:00.903 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.448366Z", 2020-11-19T03:09:00.903 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:09:00.903 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:00.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:00.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:00.904 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:09:00.904 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:00.904 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:00.904 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:09:00.904 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:00.905 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:09:00.905 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:00.905 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:09:00.905 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:09:00.906 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:00.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:00.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:00.906 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:09:00.906 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:00.907 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:00.907 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:09:00.907 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:00.907 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:09:00.907 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:00.908 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:09:00.908 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:09:00.908 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:00.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:00.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:00.909 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:09:00.909 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:00.909 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:00.909 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:09:00.909 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:00.910 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:09:00.910 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:00.910 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:09:00.910 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:09:00.910 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:00.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:00.911 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:09:00.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:00.911 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:09:00.911 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:09:00.912 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:09:00.912 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:00.912 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:09:00.913 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755280.842086_522.1 target=1_1605755195.448366_461.1 2020-11-19T03:09:01.047 INFO:tasks.rgw.c2.client.1.smithi160.stdout:2020-11-19T03:09:01.042+0000 7f69b9ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:09:05.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:09:05.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:09:05.989 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:09:06.037 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:09:06.038 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:09:06.313 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:09:06.313 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:09:06.313 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:09:06.314 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:09:06.314 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:09:06.314 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:09:06.314 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:09:06.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.315 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:09:06.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:06.315 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:09:06.315 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:06.315 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:06.316 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:09:06.316 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:06.316 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:09:06.316 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:06.316 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:09:06.317 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:09:06.317 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:06.318 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:09:06.318 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:06.318 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:06.319 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:09:06.319 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:06.319 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:09:06.319 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:06.319 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:09:06.320 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:09:06.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:06.320 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:09:06.321 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:06.321 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:06.321 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:09:06.321 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:06.321 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:09:06.322 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:06.322 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:09:06.322 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:09:06.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:06.323 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:09:06.323 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:06.323 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:06.323 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:09:06.324 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:06.324 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:09:06.324 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:06.324 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:09:06.324 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:09:06.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.325 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:09:06.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.325 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:09:06.326 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:09:06.326 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:09:06.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.326 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:09:06.327 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:09:06.328 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:09:06.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:09:06.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:09:06.329 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:09:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:09:06.373 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:09:06.622 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:09:06.622 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:09:06.623 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:09:06.623 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:09:06.623 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:09:06.623 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:09:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:09:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:09:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:09:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:09:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:09:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:09:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:09:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:09:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:09:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:09:06.626 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:09:06.626 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:09:06.626 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:09:06.626 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:09:06.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:09:06.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:09:06.627 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:09:06.627 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:09:06.627 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:09:06.628 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:09:06.628 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:09:06.628 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:09:06.628 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:09:06.628 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:09:06.629 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:09:06.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:09:06.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:09:06.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:09:06.629 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:09:06.630 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:09:06.630 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:09:06.630 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:09:06.630 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:09:06.630 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:09:06.631 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:09:06.631 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:09:06.631 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:09:06.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:09:06.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:09:06.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:09:06.632 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:09:06.632 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:09:06.633 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:09:06.633 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:09:06.633 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:09:06.633 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:09:06.633 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:09:06.634 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:09:06.634 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:09:06.634 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:09:06.634 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:09:06.634 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:09:06.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:09:06.635 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:09:06.635 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:09:06.635 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:09:06.635 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:09:06.636 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:09:06.637 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:09:06.637 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:09:06.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:09:06.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:09:06.639 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:09:06.680 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:09:06.681 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:09:06.954 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:09:06.955 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:09:06.955 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:09:06.955 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:09:06.955 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:09:06.956 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:09:06.956 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:09:06.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.956 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:09:06.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:06.957 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:09:06.957 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:06.957 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:06.957 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:09:06.958 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:06.958 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:09:06.958 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:06.958 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:09:06.958 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:09:06.959 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:06.959 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:09:06.960 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:06.960 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:06.960 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:09:06.960 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:06.960 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:09:06.961 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:06.961 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:09:06.961 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:09:06.961 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:06.962 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:09:06.964 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:06.965 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:09:06.965 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:06.966 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:09:06.966 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:09:06.966 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:06.968 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:09:06.968 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:06.969 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:09:06.969 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:09:06.969 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:09:06.970 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:09:06.970 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:09:06.971 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:09:06.971 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:09:06.971 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.972 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:09:06.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:06.973 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:09:06.973 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:09:06.975 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:09:06.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:06.976 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:09:06.977 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:09:07.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=d71482d6-15fd-4715-9992-a25958dcb538', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:09:07.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=d71482d6-15fd-4715-9992-a25958dcb538', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:09:07.006 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=d71482d6-15fd-4715-9992-a25958dcb538 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:09:07.063 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:09:07.063 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:09:07.096 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:09:07.090+0000 7fdd9866a0c0 0 failed reading obj info from .rgw.root:zone_info.d71482d6-15fd-4715-9992-a25958dcb538: (2) No such file or directory 2020-11-19T03:09:07.097 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:09:07.090+0000 7fdd9866a0c0 0 WARNING: could not read zone params for zone id=d71482d6-15fd-4715-9992-a25958dcb538 name=test-zone1 2020-11-19T03:09:07.097 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:09:07.090+0000 7fdd9866a0c0 0 failed reading obj info from .rgw.root:zone_info.fdf730f5-4390-45ac-9d84-6887e17cb661: (2) No such file or directory 2020-11-19T03:09:07.098 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:09:07.090+0000 7fdd9866a0c0 0 WARNING: could not read zone params for zone id=fdf730f5-4390-45ac-9d84-6887e17cb661 name=test-zone3 2020-11-19T03:09:07.109 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:09:07.110 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:09:07.111 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:09:07.111 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-19T03:09:07.111 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-19T03:09:07.111 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-19T03:09:07.112 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-19T03:09:07.112 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-19T03:09:07.112 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-19T03:09:07.112 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-19T03:09:07.112 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-19T03:09:07.113 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-19T03:09:07.113 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-19T03:09:07.113 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-19T03:09:07.113 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-19T03:09:07.113 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-19T03:09:07.114 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-19T03:09:07.114 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T03:09:07.114 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T03:09:07.114 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T03:09:07.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.115 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T03:09:07.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.115 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T03:09:07.115 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:09:07.115 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-19T03:09:07.116 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T03:09:07.116 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T03:09:07.116 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-19T03:09:07.116 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2020-11-19T03:09:07.116 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:07.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.117 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-19T03:09:07.117 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T03:09:07.117 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:07.118 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:07.118 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.118 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:09:07.118 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-19T03:09:07.118 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:09:07.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:09:07.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:09:07.121 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:09:07.202 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:09:07.202 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:09:07.459 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone d71482d6-15fd-4715-9992-a25958dcb538 2020-11-19T03:09:07.543 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:09:07.544 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:09:07.544 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 3, 2020-11-19T03:09:07.544 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:09:07.545 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2020-11-19T03:09:07.545 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1605755108.494399_209.1", 2020-11-19T03:09:07.545 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:09:07.545 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:09:07.545 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2020-11-19T03:09:07.546 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.546 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2020-11-19T03:09:07.546 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:09:07.546 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2020-11-19T03:09:07.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.547 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2020-11-19T03:09:07.547 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2020-11-19T03:09:07.547 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2020-11-19T03:09:07.547 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": "true", 2020-11-19T03:09:07.547 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:09:07.548 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:09:07.548 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.548 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2020-11-19T03:09:07.548 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2020-11-19T03:09:07.549 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:09:07.549 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2020-11-19T03:09:07.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.549 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:09:07.550 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2020-11-19T03:09:07.550 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:09:07.550 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:09:07.551 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.551 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:09:07.551 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:09:07.551 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:09:07.552 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:09:07.552 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2020-11-19T03:09:07.552 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "false", 2020-11-19T03:09:07.552 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2020-11-19T03:09:07.552 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2020-11-19T03:09:07.553 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.553 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:09:07.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.554 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:09:07.554 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:09:07.554 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:09:07.555 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:09:07.556 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.556 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:09:07.556 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:09:07.556 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:09:07.556 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:09:07.557 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:09:07.557 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "false", 2020-11-19T03:09:07.557 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:09:07.557 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "d71482d6-15fd-4715-9992-a25958dcb538" 2020-11-19T03:09:07.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.558 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:09:07.558 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2020-11-19T03:09:07.558 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:09:07.558 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:09:07.559 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.559 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:09:07.559 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:09:07.559 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:09:07.559 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:09:07.560 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:09:07.560 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:09:07.560 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:09:07.560 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:09:07.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.561 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:09:07.561 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2020-11-19T03:09:07.561 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:09:07.561 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:09:07.562 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.562 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:09:07.562 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:09:07.562 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:09:07.563 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:09:07.563 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:09:07.563 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:09:07.563 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:09:07.563 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:09:07.564 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:07.564 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.564 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2020-11-19T03:09:07.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.564 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2020-11-19T03:09:07.564 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2020-11-19T03:09:07.565 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2020-11-19T03:09:07.565 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2020-11-19T03:09:07.565 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:09:07.565 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:07.565 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.566 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2020-11-19T03:09:07.566 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:09:07.566 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2020-11-19T03:09:07.566 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2020-11-19T03:09:07.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:07.568 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:07.568 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:09:07.568 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2020-11-19T03:09:07.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.569 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:09:07.569 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 1446762810 2020-11-19T03:09:07.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.569 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:09:07.570 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2049584274 2020-11-19T03:09:07.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.570 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:09:07.570 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2043733610 2020-11-19T03:09:07.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:09:07.571 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:09:07.571 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2362875045 2020-11-19T03:09:07.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:07.572 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:09:07.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.572 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:09:07.572 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:09:07.572 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2020-11-19T03:09:07.573 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2020-11-19T03:09:07.573 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:09:07.573 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:09:07.573 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:09:07.574 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:09:07.574 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:09:07.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.574 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2020-11-19T03:09:07.574 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:09:07.575 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:09:07.575 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:09:07.575 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:09:07.575 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:09:07.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:09:07.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:09:07.576 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:09:07.576 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2020-11-19T03:09:07.576 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:09:07.576 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:09:07.577 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "d71482d6-15fd-4715-9992-a25958dcb538" 2020-11-19T03:10:07.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:10:08.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:10:08.305 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:10:08.352 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:10:08.353 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:10:08.382 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:10:08.377+0000 7fba28f460c0 0 failed reading obj info from .rgw.root:zone_info.d71482d6-15fd-4715-9992-a25958dcb538: (2) No such file or directory 2020-11-19T03:10:08.383 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:10:08.377+0000 7fba28f460c0 0 WARNING: could not read zone params for zone id=d71482d6-15fd-4715-9992-a25958dcb538 name=test-zone1 2020-11-19T03:10:08.383 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:10:08.377+0000 7fba28f460c0 0 failed reading obj info from .rgw.root:zone_info.fdf730f5-4390-45ac-9d84-6887e17cb661: (2) No such file or directory 2020-11-19T03:10:08.383 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:10:08.377+0000 7fba28f460c0 0 WARNING: could not read zone params for zone id=fdf730f5-4390-45ac-9d84-6887e17cb661 name=test-zone3 2020-11-19T03:10:08.393 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:10:08.393 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:10:08.394 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:10:08.394 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-19T03:10:08.394 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-19T03:10:08.395 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-19T03:10:08.395 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-19T03:10:08.395 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-19T03:10:08.395 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-19T03:10:08.395 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-19T03:10:08.396 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-19T03:10:08.396 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-19T03:10:08.396 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-19T03:10:08.396 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-19T03:10:08.396 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-19T03:10:08.397 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-19T03:10:08.397 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-19T03:10:08.397 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T03:10:08.397 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T03:10:08.397 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T03:10:08.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.398 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T03:10:08.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.398 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T03:10:08.399 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:10:08.399 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-19T03:10:08.399 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T03:10:08.399 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T03:10:08.399 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-19T03:10:08.400 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2020-11-19T03:10:08.400 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:10:08.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.401 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-19T03:10:08.401 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T03:10:08.401 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:10:08.401 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:10:08.401 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.402 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:10:08.402 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-19T03:10:08.402 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:10:08.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:10:08.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:10:08.404 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:10:08.499 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:10:08.499 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:10:08.796 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone d71482d6-15fd-4715-9992-a25958dcb538 2020-11-19T03:10:08.865 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:10:08.865 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:10:08.865 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 4, 2020-11-19T03:10:08.866 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:10:08.866 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2020-11-19T03:10:08.866 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1605755108.494399_209.1", 2020-11-19T03:10:08.866 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:10:08.867 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:10:08.867 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2020-11-19T03:10:08.867 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.867 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2020-11-19T03:10:08.867 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:10:08.868 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2020-11-19T03:10:08.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.868 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2020-11-19T03:10:08.868 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2020-11-19T03:10:08.868 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2020-11-19T03:10:08.869 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": "true", 2020-11-19T03:10:08.869 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:10:08.869 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:10:08.869 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.869 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2020-11-19T03:10:08.870 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2020-11-19T03:10:08.870 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:10:08.870 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2020-11-19T03:10:08.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.870 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:10:08.871 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2020-11-19T03:10:08.871 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:10:08.871 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:10:08.871 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.871 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:10:08.872 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:10:08.872 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:10:08.872 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:10:08.872 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2020-11-19T03:10:08.872 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "false", 2020-11-19T03:10:08.873 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2020-11-19T03:10:08.873 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2020-11-19T03:10:08.873 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.873 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:10:08.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.875 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:10:08.875 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:10:08.876 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:10:08.876 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:10:08.876 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.876 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:10:08.877 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:10:08.877 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:10:08.877 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:10:08.877 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:10:08.877 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:10:08.878 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:10:08.878 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "d71482d6-15fd-4715-9992-a25958dcb538" 2020-11-19T03:10:08.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.878 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.878 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:10:08.879 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2020-11-19T03:10:08.879 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:10:08.879 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:10:08.879 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.879 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:10:08.880 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:10:08.880 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:10:08.880 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:10:08.880 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:10:08.880 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2020-11-19T03:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2020-11-19T03:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2020-11-19T03:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2020-11-19T03:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2020-11-19T03:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2020-11-19T03:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2020-11-19T03:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2020-11-19T03:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2020-11-19T03:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2020-11-19T03:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 1446762810 2020-11-19T03:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2049584274 2020-11-19T03:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2043733610 2020-11-19T03:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2362875045 2020-11-19T03:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2020-11-19T03:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2020-11-19T03:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2020-11-19T03:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2020-11-19T03:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:10:08.906 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-11-19T03:11:08.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:11:09.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:11:09.434 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:11:09.478 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:11:09.478 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:11:09.509 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:11:09.500+0000 7f70a0fd70c0 0 failed reading obj info from .rgw.root:zone_info.d71482d6-15fd-4715-9992-a25958dcb538: (2) No such file or directory 2020-11-19T03:11:09.510 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:11:09.500+0000 7f70a0fd70c0 0 WARNING: could not read zone params for zone id=d71482d6-15fd-4715-9992-a25958dcb538 name=test-zone1 2020-11-19T03:11:09.510 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:11:09.500+0000 7f70a0fd70c0 0 failed reading obj info from .rgw.root:zone_info.fdf730f5-4390-45ac-9d84-6887e17cb661: (2) No such file or directory 2020-11-19T03:11:09.510 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:11:09.500+0000 7f70a0fd70c0 0 WARNING: could not read zone params for zone id=fdf730f5-4390-45ac-9d84-6887e17cb661 name=test-zone3 2020-11-19T03:11:09.523 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:11:09.524 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:11:09.524 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:11:09.524 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-19T03:11:09.524 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-19T03:11:09.524 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-19T03:11:09.525 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-19T03:11:09.525 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-19T03:11:09.525 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-19T03:11:09.525 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-19T03:11:09.525 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-19T03:11:09.526 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-19T03:11:09.526 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-19T03:11:09.526 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-19T03:11:09.526 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-19T03:11:09.527 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-19T03:11:09.527 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-19T03:11:09.527 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T03:11:09.527 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T03:11:09.527 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T03:11:09.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:09.528 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T03:11:09.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.528 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T03:11:09.528 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:11:09.529 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-19T03:11:09.529 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T03:11:09.529 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T03:11:09.529 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-19T03:11:09.529 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2020-11-19T03:11:09.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:11:09.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:09.530 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-19T03:11:09.530 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T03:11:09.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:11:09.531 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:11:09.531 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.531 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:11:09.531 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-19T03:11:09.531 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:11:09.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:11:09.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:11:09.534 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:11:09.613 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:11:09.613 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:11:09.899 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone d71482d6-15fd-4715-9992-a25958dcb538 2020-11-19T03:11:09.969 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:11:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:11:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 5, 2020-11-19T03:11:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:11:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2020-11-19T03:11:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1605755108.494399_209.1", 2020-11-19T03:11:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:11:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:11:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2020-11-19T03:11:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2020-11-19T03:11:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:11:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2020-11-19T03:11:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2020-11-19T03:11:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2020-11-19T03:11:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2020-11-19T03:11:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": "true", 2020-11-19T03:11:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:11:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:11:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2020-11-19T03:11:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2020-11-19T03:11:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:11:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2020-11-19T03:11:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:11:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2020-11-19T03:11:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:11:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:11:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:11:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:11:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:11:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:11:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2020-11-19T03:11:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "false", 2020-11-19T03:11:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2020-11-19T03:11:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2020-11-19T03:11:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:11:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:11:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:11:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:11:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:11:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:11:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:11:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:11:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:11:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:11:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:11:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:11:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:11:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:11:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2020-11-19T03:11:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:11:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:11:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:11:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:11:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:11:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:11:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:11:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:11:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:11:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:11:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:11:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2020-11-19T03:11:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:11:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:11:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:11:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:11:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:11:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:11:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:11:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:11:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:11:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:11:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:11:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2020-11-19T03:11:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2020-11-19T03:11:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2020-11-19T03:11:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2020-11-19T03:11:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2020-11-19T03:11:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:11:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:11:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2020-11-19T03:11:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:11:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2020-11-19T03:11:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2020-11-19T03:11:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:11:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:11:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:11:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2020-11-19T03:11:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:11:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 1446762810 2020-11-19T03:11:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:11:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2049584274 2020-11-19T03:11:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:11:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2043733610 2020-11-19T03:11:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:11:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:11:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2362875045 2020-11-19T03:11:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:11:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:11:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:11:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:11:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2020-11-19T03:11:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2020-11-19T03:11:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:11:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:11:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:11:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:11:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:11:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2020-11-19T03:11:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:11:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:11:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:11:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:11:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:11:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:11:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:11:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:11:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2020-11-19T03:11:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:11:10.005 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:11:10.006 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-11-19T03:12:10.007 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-11-19T03:12:10.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:10.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:10.029 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:10.070 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:10.071 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:10.310 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:12:10.345 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:12:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:12:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:08:00.842086Z" 2020-11-19T03:12:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:12:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:12:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:12:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:12:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:12:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:12:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:10.349 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:12:10.350 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:12:10.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:10.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:10.351 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:10.398 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:10.398 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:10.644 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:12:10.644 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:12:10.645 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:12:10.645 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:12:10.645 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:12:10.645 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:10.645 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:10.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:10.646 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:12:10.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:10.646 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:12:10.646 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:10.647 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:10.647 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:12:10.647 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:10.647 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:12:10.648 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:10.648 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:12:10.648 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:10.648 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:10.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:10.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:10.649 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:12:10.649 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:10.649 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:10.649 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:12:10.650 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:10.650 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:10.650 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:10.650 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:12:10.650 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:10.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:10.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:10.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:10.651 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:12:10.651 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:10.652 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:10.652 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:10.652 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:10.652 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:10.652 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:10.653 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:10.653 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:10.653 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:10.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:10.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:10.654 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:12:10.654 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:10.654 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:10.654 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:10.654 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:10.655 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:10.655 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:10.655 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:10.655 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:10.656 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:10.656 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:10.656 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:12:10.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:10.656 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:12:10.657 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:12:10.657 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:12:10.657 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:10.657 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:12:10.658 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:12:10.659 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:12:10.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:10.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:10.660 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:10.701 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:10.702 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:10.964 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:12:10.964 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:12:10.965 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:12:10.965 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:12:10.965 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:12:10.965 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:10.965 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:12:10.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:10.966 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:12:10.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:10.966 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:12:10.966 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:12:10.967 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:12:10.967 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:12:10.967 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:12:10.967 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:12:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:12:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:12:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:12:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:12:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:12:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:12:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:12:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:12:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:12:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:12:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:12:10.971 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:12:10.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:10.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:10.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:10.971 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:12:10.972 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:12:10.972 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:12:10.972 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:12:10.972 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:12:10.973 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:12:10.973 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:12:10.973 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:12:10.973 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:12:10.973 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:10.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:10.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:10.974 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:12:10.974 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:12:10.974 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:12:10.975 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:12:10.975 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:12:10.975 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:12:10.975 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:12:10.975 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:12:10.976 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:12:10.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:10.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:10.976 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:12:10.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:10.977 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:12:10.977 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:12:10.977 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:12:10.977 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:10.977 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:12:10.979 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:12:10.979 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:12:10.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:10.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:10.980 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:11.024 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:11.024 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:11.292 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:12:11.293 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:12:11.293 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:12:11.293 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:12:11.293 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:12:11.294 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:11.294 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:11.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:11.294 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:12:11.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:11.295 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:12:11.295 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:11.295 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:11.295 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:12:11.296 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:11.296 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:12:11.296 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:11.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:12:11.297 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:11.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:11.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:11.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:11.297 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:12:11.298 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:11.298 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:11.298 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:12:11.298 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:11.298 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:11.299 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:11.299 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:12:11.299 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:11.299 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:11.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:11.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:11.300 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:12:11.300 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:11.300 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:11.300 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:11.301 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:11.301 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:11.301 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:11.301 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:11.302 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:11.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:11.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:11.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:11.302 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:12:11.303 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:11.303 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:11.303 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:11.303 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:11.304 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:11.304 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:11.304 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:11.304 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:11.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:11.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:11.305 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:12:11.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:11.305 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:12:11.306 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:12:11.306 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:12:11.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:11.306 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:12:11.307 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:12:11.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:11.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'F8H8OCXZACJSYTWP', '--secret', 'W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:11.309 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key F8H8OCXZACJSYTWP --secret W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:11.396 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:11.397 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:11.455 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:12:11.447+0000 7ffbd21710c0 0 failed reading obj info from .rgw.root:zone_info.d71482d6-15fd-4715-9992-a25958dcb538: (2) No such file or directory 2020-11-19T03:12:11.455 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:12:11.447+0000 7ffbd21710c0 0 WARNING: could not read zone params for zone id=d71482d6-15fd-4715-9992-a25958dcb538 name=test-zone1 2020-11-19T03:12:11.456 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:12:11.447+0000 7ffbd21710c0 0 failed reading obj info from .rgw.root:zone_info.fdf730f5-4390-45ac-9d84-6887e17cb661: (2) No such file or directory 2020-11-19T03:12:11.456 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:12:11.447+0000 7ffbd21710c0 0 WARNING: could not read zone params for zone id=fdf730f5-4390-45ac-9d84-6887e17cb661 name=test-zone3 2020-11-19T03:12:11.464 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:12:11.464 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "828a9601-124a-4f83-8e27-b19c63e9fb2c", 2020-11-19T03:12:11.464 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "remove", 2020-11-19T03:12:11.465 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "remove.rgw.meta:root", 2020-11-19T03:12:11.465 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "remove.rgw.control", 2020-11-19T03:12:11.465 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "remove.rgw.log:gc", 2020-11-19T03:12:11.465 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "remove.rgw.log:lc", 2020-11-19T03:12:11.465 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "remove.rgw.log", 2020-11-19T03:12:11.466 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-11-19T03:12:11.466 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-11-19T03:12:11.466 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-11-19T03:12:11.466 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-11-19T03:12:11.466 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-11-19T03:12:11.467 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-11-19T03:12:11.467 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-11-19T03:12:11.467 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-11-19T03:12:11.467 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "remove.rgw.otp", 2020-11-19T03:12:11.467 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2020-11-19T03:12:11.468 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "F8H8OCXZACJSYTWP", 2020-11-19T03:12:11.468 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "W0vgRNhwEWo9YuYAj4GZXaAHYh6trUyS" 2020-11-19T03:12:11.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:11.468 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2020-11-19T03:12:11.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:11.469 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2020-11-19T03:12:11.469 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:11.471 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "remove.rgw.buckets.index", 2020-11-19T03:12:11.471 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2020-11-19T03:12:11.471 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2020-11-19T03:12:11.472 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "remove.rgw.buckets.data" 2020-11-19T03:12:11.472 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:11.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:11.472 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-11-19T03:12:11.473 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2020-11-19T03:12:11.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:11.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:11.473 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:11.473 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:11.474 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "remove.rgw.log:notif" 2020-11-19T03:12:11.474 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:12:11.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:11.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:11.476 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:11.515 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:11.515 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:19.598 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone d71482d6-15fd-4715-9992-a25958dcb538 2020-11-19T03:12:19.679 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:12:19.679 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:19.679 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 6, 2020-11-19T03:12:19.679 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:12:19.680 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2020-11-19T03:12:19.680 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1605755108.494399_209.1", 2020-11-19T03:12:19.680 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:12:19.680 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2020-11-19T03:12:19.680 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2020-11-19T03:12:19.681 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.681 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2020-11-19T03:12:19.681 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:19.681 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2020-11-19T03:12:19.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.682 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2020-11-19T03:12:19.682 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2020-11-19T03:12:19.682 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2020-11-19T03:12:19.682 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": "true", 2020-11-19T03:12:19.683 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:12:19.683 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:19.683 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.683 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2020-11-19T03:12:19.683 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2020-11-19T03:12:19.684 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:19.684 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2020-11-19T03:12:19.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.684 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:12:19.684 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2020-11-19T03:12:19.685 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:12:19.685 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:12:19.685 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.685 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:12:19.685 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:12:19.686 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:19.686 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:12:19.686 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2020-11-19T03:12:19.686 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "false", 2020-11-19T03:12:19.687 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2020-11-19T03:12:19.687 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2020-11-19T03:12:19.687 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.687 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:12:19.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.688 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "828a9601-124a-4f83-8e27-b19c63e9fb2c", 2020-11-19T03:12:19.688 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "remove", 2020-11-19T03:12:19.688 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [], 2020-11-19T03:12:19.688 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:12:19.689 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:12:19.689 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:19.689 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:12:19.689 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:12:19.690 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:12:19.690 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:12:19.690 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:12:19.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.691 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:12:19.691 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2020-11-19T03:12:19.691 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:12:19.691 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:12:19.691 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.692 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:12:19.692 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:12:19.692 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:19.692 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:12:19.692 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:12:19.693 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:12:19.693 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:12:19.693 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:12:19.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.694 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:19.694 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2020-11-19T03:12:19.694 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:12:19.694 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:19.694 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.695 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:12:19.695 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:12:19.695 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:19.695 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:12:19.695 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:12:19.696 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:12:19.696 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:12:19.696 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:12:19.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.697 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:12:19.697 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2020-11-19T03:12:19.697 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2020-11-19T03:12:19.697 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:12:19.697 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.698 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": "false", 2020-11-19T03:12:19.698 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": "true", 2020-11-19T03:12:19.698 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:19.698 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": "false", 2020-11-19T03:12:19.698 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2020-11-19T03:12:19.699 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": "true", 2020-11-19T03:12:19.699 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2020-11-19T03:12:19.699 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "" 2020-11-19T03:12:19.699 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:19.708 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.709 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2020-11-19T03:12:19.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.709 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2020-11-19T03:12:19.709 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2020-11-19T03:12:19.710 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2020-11-19T03:12:19.710 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2020-11-19T03:12:19.710 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:12:19.710 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:19.711 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.711 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2020-11-19T03:12:19.711 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:19.711 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2020-11-19T03:12:19.711 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2020-11-19T03:12:19.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:19.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:19.712 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2020-11-19T03:12:19.712 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2020-11-19T03:12:19.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.713 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:12:19.713 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 1446762810 2020-11-19T03:12:19.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.713 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "828a9601-124a-4f83-8e27-b19c63e9fb2c", 2020-11-19T03:12:19.714 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 1008807272 2020-11-19T03:12:19.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.714 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:12:19.714 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2049584274 2020-11-19T03:12:19.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.715 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:19.715 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2043733610 2020-11-19T03:12:19.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:19.716 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:12:19.716 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2362875045 2020-11-19T03:12:19.716 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:19.716 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:12:19.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.717 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:12:19.717 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:19.717 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2020-11-19T03:12:19.718 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2020-11-19T03:12:19.718 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:12:19.718 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:12:19.718 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:12:19.718 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:12:19.719 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:12:19.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.719 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2020-11-19T03:12:19.719 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2020-11-19T03:12:19.719 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2020-11-19T03:12:19.720 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2020-11-19T03:12:19.720 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2020-11-19T03:12:19.720 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2020-11-19T03:12:19.720 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:19.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:19.721 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:19.721 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2020-11-19T03:12:19.721 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:19.721 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:12:19.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:19.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:19.724 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:19.782 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:12:19.776+0000 7f632d41f0c0 0 failed reading obj info from .rgw.root:zone_info.0c377913-e2b8-4ccf-b637-8d38a8727f98: (2) No such file or directory 2020-11-19T03:12:19.782 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:12:19.776+0000 7f632d41f0c0 0 WARNING: could not read zone params for zone id=0c377913-e2b8-4ccf-b637-8d38a8727f98 name=test-zone4 2020-11-19T03:12:19.783 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:12:19.776+0000 7f632d41f0c0 0 failed reading obj info from .rgw.root:zone_info.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c: (2) No such file or directory 2020-11-19T03:12:19.783 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:12:19.776+0000 7f632d41f0c0 0 WARNING: could not read zone params for zone id=c9a90d7d-dd43-4d9d-9b12-b4d640ca300c name=test-zone2 2020-11-19T03:12:19.793 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:12:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "test-zonegroup", 2020-11-19T03:12:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zonegroup", 2020-11-19T03:12:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "", 2020-11-19T03:12:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "true", 2020-11-19T03:12:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T03:12:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T03:12:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [ 2020-11-19T03:12:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:12:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone4", 2020-11-19T03:12:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:12:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "pubsub", 2020-11-19T03:12:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "false", 2020-11-19T03:12:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [ 2020-11-19T03:12:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: "test-zone1" 2020-11-19T03:12:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:19.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:19.800 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:12:19.800 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone2", 2020-11-19T03:12:19.800 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:19.800 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:12:19.801 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:19.801 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:19.801 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:19.801 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:19.802 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:19.802 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:19.802 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:19.802 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:19.802 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:19.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:19.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:19.803 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:19.803 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T03:12:19.803 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:19.804 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:19.804 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:19.804 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:19.804 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:19.804 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:19.805 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:19.805 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:19.805 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:19.805 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:19.806 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:19.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:19.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:19.806 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:12:19.807 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone3", 2020-11-19T03:12:19.807 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:19.807 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:12:19.807 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:19.808 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:19.808 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:19.808 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:19.808 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:19.808 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:19.809 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:19.809 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:19.809 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:19.809 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:19.809 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:19.810 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [ 2020-11-19T03:12:19.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:19.810 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default-placement", 2020-11-19T03:12:19.810 INFO:teuthology.orchestra.run.smithi086.stdout: "tags": [], 2020-11-19T03:12:19.810 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": [ 2020-11-19T03:12:19.811 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD" 2020-11-19T03:12:19.811 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:12:19.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:19.811 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:19.811 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "default-placement", 2020-11-19T03:12:19.812 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:19.812 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T03:12:19.812 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T03:12:19.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:19.812 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:12:19.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:19.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:19.815 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:19.894 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:19.895 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:19.914 INFO:teuthology.orchestra.run.smithi086.stderr:zone name remove not found in zonegroup test-zonegroup 2020-11-19T03:12:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-11-19T03:12:19.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:19.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:19.922 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:19.961 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:19.961 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:20.005 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:12:19.995+0000 7ff60493d0c0 0 failed reading obj info from .rgw.root:zone_info.d71482d6-15fd-4715-9992-a25958dcb538: (2) No such file or directory 2020-11-19T03:12:20.006 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:12:19.995+0000 7ff60493d0c0 0 WARNING: could not read zone params for zone id=d71482d6-15fd-4715-9992-a25958dcb538 name=test-zone1 2020-11-19T03:12:20.006 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:12:19.995+0000 7ff60493d0c0 0 failed reading obj info from .rgw.root:zone_info.fdf730f5-4390-45ac-9d84-6887e17cb661: (2) No such file or directory 2020-11-19T03:12:20.006 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:12:19.995+0000 7ff60493d0c0 0 WARNING: could not read zone params for zone id=fdf730f5-4390-45ac-9d84-6887e17cb661 name=test-zone3 2020-11-19T03:12:20.010 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:12:20.003+0000 7ff60493d0c0 0 zone id 828a9601-124a-4f83-8e27-b19c63e9fb2c is not a part of zonegroup default 2020-11-19T03:12:20.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:20.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:20.028 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:20.075 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:20.076 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:20.161 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:12:20.156+0000 7f73d7c8f0c0 0 WARNING: can't generate connection for zone 828a9601-124a-4f83-8e27-b19c63e9fb2c id remove: no endpoints defined 2020-11-19T03:12:20.572 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:12:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 7, 2020-11-19T03:12:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: "predecessor_uuid": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:12:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": [ 2020-11-19T03:12:20.573 INFO:teuthology.orchestra.run.smithi086.stdout: "1_1605755108.494399_209.1", 2020-11-19T03:12:20.574 INFO:teuthology.orchestra.run.smithi086.stdout: "", 2020-11-19T03:12:20.574 INFO:teuthology.orchestra.run.smithi086.stdout: "", 2020-11-19T03:12:20.574 INFO:teuthology.orchestra.run.smithi086.stdout: "" 2020-11-19T03:12:20.574 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: "period_map": { 2020-11-19T03:12:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: "zonegroups": [ 2020-11-19T03:12:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "test-zonegroup", 2020-11-19T03:12:20.576 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zonegroup", 2020-11-19T03:12:20.576 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "", 2020-11-19T03:12:20.576 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "true", 2020-11-19T03:12:20.576 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:20.577 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:20.577 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.577 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T03:12:20.577 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T03:12:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [ 2020-11-19T03:12:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:12:20.578 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone4", 2020-11-19T03:12:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:12:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:20.579 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:20.580 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:20.580 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:20.580 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "pubsub", 2020-11-19T03:12:20.580 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "false", 2020-11-19T03:12:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [ 2020-11-19T03:12:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "test-zone1" 2020-11-19T03:12:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:12:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone2", 2020-11-19T03:12:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:20.582 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:12:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:20.583 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:20.584 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:20.584 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:20.584 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:20.584 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T03:12:20.585 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:20.586 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:20.587 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:20.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.588 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:12:20.588 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone3", 2020-11-19T03:12:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:12:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:20.589 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:20.590 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:20.590 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:20.590 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:20.590 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.591 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [ 2020-11-19T03:12:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default-placement", 2020-11-19T03:12:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: "tags": [], 2020-11-19T03:12:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": [ 2020-11-19T03:12:20.592 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD" 2020-11-19T03:12:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:12:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "default-placement", 2020-11-19T03:12:20.593 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:20.594 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T03:12:20.594 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T03:12:20.594 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.594 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.595 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.616 INFO:teuthology.orchestra.run.smithi086.stdout: "short_zone_ids": [ 2020-11-19T03:12:20.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.616 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:12:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 1446762810 2020-11-19T03:12:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:12:20.617 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2049584274 2020-11-19T03:12:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2043733610 2020-11-19T03:12:20.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:12:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2362875045 2020-11-19T03:12:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.619 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:12:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:12:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:20.620 INFO:teuthology.orchestra.run.smithi086.stdout: "period_config": { 2020-11-19T03:12:20.621 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2020-11-19T03:12:20.621 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T03:12:20.621 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T03:12:20.621 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T03:12:20.621 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T03:12:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T03:12:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2020-11-19T03:12:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T03:12:20.622 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T03:12:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T03:12:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T03:12:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T03:12:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.624 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:20.624 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_name": "test-realm", 2020-11-19T03:12:20.624 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:12:20.624 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:12:20.625 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-11-19T03:12:20.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:20.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:20.627 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:20.665 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:20.665 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:20.694 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:12:20.694 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "73c25c0a-ceaa-4c88-88de-f38d6ee61ef9", 2020-11-19T03:12:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "remove_zg", 2020-11-19T03:12:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "remove_zg", 2020-11-19T03:12:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "false", 2020-11-19T03:12:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [], 2020-11-19T03:12:20.696 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T03:12:20.696 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T03:12:20.696 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "", 2020-11-19T03:12:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [], 2020-11-19T03:12:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [], 2020-11-19T03:12:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "", 2020-11-19T03:12:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:20.697 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T03:12:20.698 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T03:12:20.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.698 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:12:20.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:20.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:20.700 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:20.785 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:20.785 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:20.866 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:12:20.866 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "c3a868b9-964e-49c7-8c42-17d26ec6eb7e", 2020-11-19T03:12:20.866 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "remove", 2020-11-19T03:12:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "domain_root": "remove.rgw.meta:root", 2020-11-19T03:12:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "control_pool": "remove.rgw.control", 2020-11-19T03:12:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "gc_pool": "remove.rgw.log:gc", 2020-11-19T03:12:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "lc_pool": "remove.rgw.log:lc", 2020-11-19T03:12:20.867 INFO:teuthology.orchestra.run.smithi086.stdout: "log_pool": "remove.rgw.log", 2020-11-19T03:12:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-11-19T03:12:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-11-19T03:12:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-11-19T03:12:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-11-19T03:12:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-11-19T03:12:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-11-19T03:12:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-11-19T03:12:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-11-19T03:12:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: "otp_pool": "remove.rgw.otp", 2020-11-19T03:12:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: "system_key": { 2020-11-19T03:12:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "", 2020-11-19T03:12:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "" 2020-11-19T03:12:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_pools": [ 2020-11-19T03:12:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "default-placement", 2020-11-19T03:12:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:12:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "index_pool": "remove.rgw.buckets.index", 2020-11-19T03:12:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": { 2020-11-19T03:12:20.871 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD": { 2020-11-19T03:12:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: "data_pool": "remove.rgw.buckets.data" 2020-11-19T03:12:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-11-19T03:12:20.872 INFO:teuthology.orchestra.run.smithi086.stdout: "index_type": 0 2020-11-19T03:12:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:20.873 INFO:teuthology.orchestra.run.smithi086.stdout: "notif_pool": "remove.rgw.log:notif" 2020-11-19T03:12:20.874 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:12:20.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:20.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:20.876 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:20.962 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:20.962 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:21.312 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:12:21.313 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:21.313 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 8, 2020-11-19T03:12:21.313 INFO:teuthology.orchestra.run.smithi086.stdout: "predecessor_uuid": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:12:21.313 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": [ 2020-11-19T03:12:21.314 INFO:teuthology.orchestra.run.smithi086.stdout: "1_1605755108.494399_209.1", 2020-11-19T03:12:21.314 INFO:teuthology.orchestra.run.smithi086.stdout: "", 2020-11-19T03:12:21.314 INFO:teuthology.orchestra.run.smithi086.stdout: "", 2020-11-19T03:12:21.314 INFO:teuthology.orchestra.run.smithi086.stdout: "" 2020-11-19T03:12:21.315 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.315 INFO:teuthology.orchestra.run.smithi086.stdout: "period_map": { 2020-11-19T03:12:21.315 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:21.315 INFO:teuthology.orchestra.run.smithi086.stdout: "zonegroups": [ 2020-11-19T03:12:21.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.316 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "73c25c0a-ceaa-4c88-88de-f38d6ee61ef9", 2020-11-19T03:12:21.316 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "remove_zg", 2020-11-19T03:12:21.316 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "remove_zg", 2020-11-19T03:12:21.316 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "false", 2020-11-19T03:12:21.317 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [], 2020-11-19T03:12:21.317 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T03:12:21.317 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T03:12:21.317 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "c3a868b9-964e-49c7-8c42-17d26ec6eb7e", 2020-11-19T03:12:21.318 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [ 2020-11-19T03:12:21.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.318 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "c3a868b9-964e-49c7-8c42-17d26ec6eb7e", 2020-11-19T03:12:21.318 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "remove", 2020-11-19T03:12:21.319 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [], 2020-11-19T03:12:21.319 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:21.319 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "false", 2020-11-19T03:12:21.319 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:21.320 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:21.320 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:21.320 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:21.320 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:21.321 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:21.321 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:21.321 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.322 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [ 2020-11-19T03:12:21.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.323 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default-placement", 2020-11-19T03:12:21.323 INFO:teuthology.orchestra.run.smithi086.stdout: "tags": [], 2020-11-19T03:12:21.323 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": [ 2020-11-19T03:12:21.323 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD" 2020-11-19T03:12:21.324 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:12:21.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:21.324 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.324 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "default-placement", 2020-11-19T03:12:21.325 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:21.325 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T03:12:21.325 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T03:12:21.325 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:21.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.326 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "test-zonegroup", 2020-11-19T03:12:21.326 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zonegroup", 2020-11-19T03:12:21.326 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "", 2020-11-19T03:12:21.326 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "true", 2020-11-19T03:12:21.327 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:21.327 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:21.327 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.327 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T03:12:21.327 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T03:12:21.328 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:21.328 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [ 2020-11-19T03:12:21.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.328 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:12:21.329 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone4", 2020-11-19T03:12:21.329 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:21.329 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:12:21.329 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.329 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:21.330 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:21.330 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:21.330 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:21.330 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "pubsub", 2020-11-19T03:12:21.330 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "false", 2020-11-19T03:12:21.331 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [ 2020-11-19T03:12:21.331 INFO:teuthology.orchestra.run.smithi086.stdout: "test-zone1" 2020-11-19T03:12:21.331 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.331 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:21.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.332 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:12:21.332 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone2", 2020-11-19T03:12:21.332 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:21.333 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:12:21.333 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.333 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:21.333 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:21.333 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:21.334 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:21.334 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:21.334 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:21.334 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:21.334 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:21.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.335 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:21.335 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T03:12:21.336 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:21.336 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:21.336 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.336 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:21.336 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:21.337 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:21.338 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:21.338 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:21.338 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:21.338 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:21.339 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:21.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.339 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:12:21.339 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone3", 2020-11-19T03:12:21.340 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:21.340 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:12:21.340 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.340 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:21.340 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:21.341 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:21.341 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:21.341 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:21.341 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:21.341 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:21.342 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:21.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:21.342 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.342 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [ 2020-11-19T03:12:21.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.343 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default-placement", 2020-11-19T03:12:21.343 INFO:teuthology.orchestra.run.smithi086.stdout: "tags": [], 2020-11-19T03:12:21.343 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": [ 2020-11-19T03:12:21.343 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD" 2020-11-19T03:12:21.343 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:12:21.344 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:21.344 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.344 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "default-placement", 2020-11-19T03:12:21.345 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:21.345 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T03:12:21.345 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T03:12:21.345 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:21.345 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:21.346 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:21.346 INFO:teuthology.orchestra.run.smithi086.stdout: "short_zone_ids": [ 2020-11-19T03:12:21.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.346 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:12:21.346 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 1446762810 2020-11-19T03:12:21.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.347 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "c3a868b9-964e-49c7-8c42-17d26ec6eb7e", 2020-11-19T03:12:21.347 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 4028195125 2020-11-19T03:12:21.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.348 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:12:21.348 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2049584274 2020-11-19T03:12:21.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.349 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:21.349 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2043733610 2020-11-19T03:12:21.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:21.349 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:12:21.350 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2362875045 2020-11-19T03:12:21.350 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:21.350 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:12:21.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.351 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:12:21.351 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:21.351 INFO:teuthology.orchestra.run.smithi086.stdout: "period_config": { 2020-11-19T03:12:21.351 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2020-11-19T03:12:21.351 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T03:12:21.352 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T03:12:21.352 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T03:12:21.352 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T03:12:21.352 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T03:12:21.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.353 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2020-11-19T03:12:21.353 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T03:12:21.353 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T03:12:21.353 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T03:12:21.353 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T03:12:21.354 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T03:12:21.354 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:21.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:21.354 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:21.354 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_name": "test-realm", 2020-11-19T03:12:21.355 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:12:21.355 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:12:21.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:21.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:21.356 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:21.475 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:12:21.468+0000 7f77b24a60c0 0 zone id c3a868b9-964e-49c7-8c42-17d26ec6eb7e is not a part of zonegroup test-zonegroup 2020-11-19T03:12:21.476 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:12:21.468+0000 7f77b24a60c0 0 zone id c3a868b9-964e-49c7-8c42-17d26ec6eb7e is not a part of zonegroup default 2020-11-19T03:12:21.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:21.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:21.496 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:21.575 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:21.575 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:21.616 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:12:21.608+0000 7f89e6f1d0c0 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-11-19T03:12:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-11-19T03:12:21.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:21.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:21.625 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:21.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:21.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:12:21.745 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:12:21.835 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:21.836 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:22.142 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:12:22.142 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:22.143 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 9, 2020-11-19T03:12:22.143 INFO:teuthology.orchestra.run.smithi086.stdout: "predecessor_uuid": "0e78ea22-1fd1-48d9-9be5-2a207091b0ca", 2020-11-19T03:12:22.143 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": [ 2020-11-19T03:12:22.143 INFO:teuthology.orchestra.run.smithi086.stdout: "1_1605755108.494399_209.1", 2020-11-19T03:12:22.143 INFO:teuthology.orchestra.run.smithi086.stdout: "", 2020-11-19T03:12:22.144 INFO:teuthology.orchestra.run.smithi086.stdout: "", 2020-11-19T03:12:22.144 INFO:teuthology.orchestra.run.smithi086.stdout: "" 2020-11-19T03:12:22.144 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.144 INFO:teuthology.orchestra.run.smithi086.stdout: "period_map": { 2020-11-19T03:12:22.144 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:22.145 INFO:teuthology.orchestra.run.smithi086.stdout: "zonegroups": [ 2020-11-19T03:12:22.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.145 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "test-zonegroup", 2020-11-19T03:12:22.145 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zonegroup", 2020-11-19T03:12:22.145 INFO:teuthology.orchestra.run.smithi086.stdout: "api_name": "", 2020-11-19T03:12:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: "is_master": "true", 2020-11-19T03:12:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames": [], 2020-11-19T03:12:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "hostnames_s3website": [], 2020-11-19T03:12:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "zones": [ 2020-11-19T03:12:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:12:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone4", 2020-11-19T03:12:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2020-11-19T03:12:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "pubsub", 2020-11-19T03:12:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "false", 2020-11-19T03:12:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [ 2020-11-19T03:12:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "test-zone1" 2020-11-19T03:12:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:12:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone2", 2020-11-19T03:12:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2020-11-19T03:12:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone1", 2020-11-19T03:12:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8000" 2020-11-19T03:12:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "id": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:12:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "test-zone3", 2020-11-19T03:12:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "endpoints": [ 2020-11-19T03:12:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "http://smithi086.front.sepia.ceph.com:8001" 2020-11-19T03:12:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "log_meta": "false", 2020-11-19T03:12:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "log_data": "true", 2020-11-19T03:12:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:12:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "read_only": "false", 2020-11-19T03:12:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "tier_type": "", 2020-11-19T03:12:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from_all": "true", 2020-11-19T03:12:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_from": [], 2020-11-19T03:12:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "redirect_zone": "" 2020-11-19T03:12:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_targets": [ 2020-11-19T03:12:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "default-placement", 2020-11-19T03:12:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: "tags": [], 2020-11-19T03:12:22.161 INFO:teuthology.orchestra.run.smithi086.stdout: "storage_classes": [ 2020-11-19T03:12:22.161 INFO:teuthology.orchestra.run.smithi086.stdout: "STANDARD" 2020-11-19T03:12:22.161 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:12:22.161 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:22.161 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.162 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "default-placement", 2020-11-19T03:12:22.162 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:22.162 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_policy": { 2020-11-19T03:12:22.162 INFO:teuthology.orchestra.run.smithi086.stdout: "groups": [] 2020-11-19T03:12:22.162 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:22.163 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:22.163 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2020-11-19T03:12:22.174 INFO:teuthology.orchestra.run.smithi086.stdout: "short_zone_ids": [ 2020-11-19T03:12:22.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.175 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "0c377913-e2b8-4ccf-b637-8d38a8727f98", 2020-11-19T03:12:22.175 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 1446762810 2020-11-19T03:12:22.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:22.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.176 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:12:22.176 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2049584274 2020-11-19T03:12:22.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:22.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.177 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:22.177 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2043733610 2020-11-19T03:12:22.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:22.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:22.177 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:12:22.178 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 2362875045 2020-11-19T03:12:22.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:22.178 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:12:22.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:22.178 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:12:22.179 INFO:teuthology.orchestra.run.smithi086.stdout: "master_zone": "d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:12:22.179 INFO:teuthology.orchestra.run.smithi086.stdout: "period_config": { 2020-11-19T03:12:22.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2020-11-19T03:12:22.179 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T03:12:22.179 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T03:12:22.180 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T03:12:22.180 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T03:12:22.180 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T03:12:22.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:22.180 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2020-11-19T03:12:22.181 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2020-11-19T03:12:22.181 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2020-11-19T03:12:22.181 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2020-11-19T03:12:22.181 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2020-11-19T03:12:22.181 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2020-11-19T03:12:22.182 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:22.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:22.182 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_id": "46bd204f-1547-4756-ae4a-a8264d05dd86", 2020-11-19T03:12:22.182 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_name": "test-realm", 2020-11-19T03:12:22.182 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:12:22.183 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:12:22.184 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-11-19T03:12:22.185 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-50 2020-11-19T03:12:25.908 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-51 2020-11-19T03:12:31.452 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-52 2020-11-19T03:12:31.525 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-53 2020-11-19T03:12:31.564 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-54 2020-11-19T03:12:31.643 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-55 2020-11-19T03:12:33.933 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-11-19T03:12:35.296 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-56 2020-11-19T03:12:35.339 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-57 2020-11-19T03:12:35.403 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-58 2020-11-19T03:12:35.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:35.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:35.471 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:35.509 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:12:35.771 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:12:35.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:35.771 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755555.445286_715.1", 2020-11-19T03:12:35.771 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:12:35.445286Z" 2020-11-19T03:12:35.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:35.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:35.772 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:12:35.772 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:12:35.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:35.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:35.773 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:12:35.773 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:12:35.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:12:35.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:12:35.774 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:12:35.774 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:12:35.774 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:12:35.774 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:12:35.775 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:12:35.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:35.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:35.777 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:35.816 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:35.817 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:36.073 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:12:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:12:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:12:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:12:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:12:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:12:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:12:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:12:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:12:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:12:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:12:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:12:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:12:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:36.081 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:12:36.081 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:36.081 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:36.081 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:36.081 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:36.082 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:36.082 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:36.082 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:36.082 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:36.082 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:12:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:36.084 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:36.084 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:36.084 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:36.084 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:36.084 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:36.085 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:36.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:36.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:36.085 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:12:36.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:36.086 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:12:36.086 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:12:36.086 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:12:36.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:36.086 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:12:36.088 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:12:41.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:41.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:41.090 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:41.133 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:41.134 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:41.393 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:12:41.393 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:12:41.394 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:12:41.394 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:12:41.394 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:12:41.394 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:41.395 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:41.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:41.395 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:12:41.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:41.395 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:12:41.396 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:41.396 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:41.396 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:12:41.396 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:41.396 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:12:41.397 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:41.397 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:12:41.397 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:41.397 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:41.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:41.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:41.398 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:12:41.398 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:41.398 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:41.398 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:12:41.399 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:41.399 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:41.399 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:41.399 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:12:41.399 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:41.400 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:41.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:41.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:41.400 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:12:41.401 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:41.401 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:41.401 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:41.401 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:41.401 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:41.402 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:41.402 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:41.402 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:41.402 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:41.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:41.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:41.403 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:12:41.403 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:41.403 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:41.403 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:41.404 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:41.404 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:41.404 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:41.404 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:41.404 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:41.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:41.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:41.405 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:12:41.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:41.405 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:12:41.406 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:12:41.406 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:12:41.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:41.406 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:12:41.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:12:46.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:46.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:46.410 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:46.450 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:46.451 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:46.730 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:12:46.730 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:12:46.730 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:12:46.731 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:12:46.731 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:12:46.731 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:46.731 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:46.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:46.732 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:12:46.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:46.732 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:12:46.732 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:46.732 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:46.733 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:12:46.733 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:46.733 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:12:46.733 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:46.733 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:12:46.734 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:46.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:46.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:46.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:46.735 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:12:46.735 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:46.735 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:46.735 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:12:46.735 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:46.736 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:46.736 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:46.736 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:12:46.736 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:46.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:46.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:46.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:46.737 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:12:46.737 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:46.737 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:46.738 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:46.738 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:46.738 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:46.738 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:46.738 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:46.739 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:46.739 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:46.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:46.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:46.739 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:12:46.740 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:46.740 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:46.740 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:46.740 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:46.740 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:46.741 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:46.741 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:46.741 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:46.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:46.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:46.742 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:12:46.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:46.742 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:12:46.742 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:12:46.742 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:12:46.743 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:46.743 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:12:46.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:12:51.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:51.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:51.746 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:51.788 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:51.788 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:52.066 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:12:52.066 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:12:52.067 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:12:52.067 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:12:52.067 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:12:52.067 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:52.068 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:52.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:52.068 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:12:52.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:52.069 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:12:52.069 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:52.069 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:52.069 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:12:52.069 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:52.070 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:12:52.070 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:52.070 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:12:52.070 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:52.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:52.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:52.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:52.071 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:12:52.071 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:52.072 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:52.072 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:12:52.072 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:52.072 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:52.072 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:52.073 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:12:52.073 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:52.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:52.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:52.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:52.074 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:12:52.074 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:52.074 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:52.074 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:52.075 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:52.075 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:52.076 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:52.076 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:52.076 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:52.076 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:52.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:52.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:52.077 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:12:52.077 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:52.077 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:52.077 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:52.078 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:52.078 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:52.078 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:52.078 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:52.079 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:52.079 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:52.079 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:52.079 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:12:52.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:52.080 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:12:52.080 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:12:52.080 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:12:52.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:52.081 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:12:52.082 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:12:57.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:57.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:12:57.469 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:12:57.511 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:12:57.511 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:12:57.792 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:12:57.793 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:12:57.793 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:12:57.793 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:12:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:12:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:12:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:12:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:12:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:12:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:12:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:12:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:12:57.797 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:57.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:57.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:57.797 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:12:57.797 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:12:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:12:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:57.800 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:12:57.800 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:57.800 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:57.802 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:57.802 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:57.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:57.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:12:57.802 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:12:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:12:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:12:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:12:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:12:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:12:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:12:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:12:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:12:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:57.805 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:12:57.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:12:57.805 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:12:57.805 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:12:57.805 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:12:57.806 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:12:57.806 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:12:57.807 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:02.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:02.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:02.810 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:02.851 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:02.851 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:03.117 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:03.119 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:03.119 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:03.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:03.119 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:03.126 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:03.126 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:03.126 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:03.126 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:03.127 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:03.127 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:03.127 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:03.127 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:03.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:03.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:03.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:03.128 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:03.128 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:03.132 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:03.132 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:03.132 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:03.133 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:08.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:09.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:09.181 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:09.224 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:09.224 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:09.492 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:09.492 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:09.493 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:09.493 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:09.494 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:09.494 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:09.494 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:09.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:09.495 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:09.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:09.495 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:09.495 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:09.496 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:09.496 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:09.496 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:09.496 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:09.497 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:09.497 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:09.498 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:09.498 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:09.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:09.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:09.499 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:09.499 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:09.499 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:09.500 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:09.500 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:09.500 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:09.500 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:09.501 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:09.501 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:09.501 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:09.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:09.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:09.502 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:09.503 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:09.503 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:09.503 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:09.503 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:09.504 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:09.504 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:09.504 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:09.504 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:09.505 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:09.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:09.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:09.506 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:09.506 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:09.507 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:09.507 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:09.507 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:09.507 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:09.507 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:09.508 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:09.508 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:09.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:09.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:09.508 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:09.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:09.509 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:09.509 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:09.509 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:09.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:09.510 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:09.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:14.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:14.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:14.513 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:14.557 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:14.558 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:14.848 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:14.849 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:14.849 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:14.849 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:14.850 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:14.850 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:14.850 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:14.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:14.850 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:14.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:14.851 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:14.851 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:14.852 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:14.852 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:14.852 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:14.852 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:14.853 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:14.853 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:14.853 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:14.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:14.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:14.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:14.854 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:14.854 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:14.854 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:14.855 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:14.855 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:14.855 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:14.856 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:14.857 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:14.857 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:14.857 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:14.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:14.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:14.858 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:14.858 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:14.859 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:14.859 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:14.859 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:14.859 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:14.860 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:14.860 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:14.860 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:14.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:14.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:14.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:14.861 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:14.861 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:14.862 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:14.862 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:14.862 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:14.862 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:14.862 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:14.863 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:14.863 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:14.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:14.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:14.863 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:14.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:14.864 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:14.864 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:14.865 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:14.865 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:14.865 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:14.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:19.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:19.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:19.992 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:20.035 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:20.036 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:20.301 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:20.301 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:20.302 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:20.302 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:20.302 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:20.302 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:20.302 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:20.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:20.303 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:20.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:20.303 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:20.304 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:20.304 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:20.304 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:20.304 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:20.304 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:20.305 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:20.305 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:20.305 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:20.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:20.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:20.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:20.306 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:20.306 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:20.306 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:20.307 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:20.307 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:20.307 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:20.307 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:20.307 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:20.308 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:20.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:20.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:20.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:20.308 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:20.309 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:20.309 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:20.309 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:20.309 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:20.309 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:20.310 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:20.310 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:20.310 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:20.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:20.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:20.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:20.311 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:20.311 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:20.311 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:20.312 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:20.312 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:20.312 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:20.312 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:20.312 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:20.313 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:20.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:20.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:20.313 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:20.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:20.314 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:20.314 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:20.314 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:20.314 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:20.314 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:20.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:25.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:25.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:25.320 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:25.364 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:25.365 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:25.630 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:25.630 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:25.631 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:25.631 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:25.631 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:25.631 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:25.631 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:25.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:25.632 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:25.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:25.632 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:25.632 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:25.633 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:25.633 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:25.633 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:25.633 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:25.634 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:25.634 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:25.634 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:25.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:25.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:25.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:25.636 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:25.636 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:25.636 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:25.636 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:25.636 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:25.637 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:25.637 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:25.637 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:25.637 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:25.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:25.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:25.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:25.638 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:25.638 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:25.639 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:25.639 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:25.639 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:25.640 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:25.640 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:25.640 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:25.640 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:25.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:25.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:25.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:25.641 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:25.642 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:25.642 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:25.642 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:25.642 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:25.642 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:25.643 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:25.643 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:25.643 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:25.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:25.644 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:25.644 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:25.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:25.644 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:25.644 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:25.645 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:25.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:25.645 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:25.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:30.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:30.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:30.648 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:30.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:30.693 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:30.976 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:30.976 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:30.977 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:30.977 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:30.977 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:30.977 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:30.978 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:30.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:30.978 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:30.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:30.978 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:30.979 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:30.979 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:30.979 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:30.979 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:30.979 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:30.980 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:30.980 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:30.980 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:30.980 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:30.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:30.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:30.981 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:30.981 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:30.981 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:30.982 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:30.982 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:30.982 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:30.982 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:30.982 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:30.983 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:30.983 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:30.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:30.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:30.983 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:30.984 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:30.984 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:30.984 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:30.984 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:30.984 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:30.985 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:30.985 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:30.985 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:30.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:30.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:30.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:30.986 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:30.986 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:30.986 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:30.987 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:30.987 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:30.987 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:30.987 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:30.987 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:30.988 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:30.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:30.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:30.988 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:30.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:30.989 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:30.989 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:30.989 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:30.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:30.990 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:30.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:35.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:35.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:35.994 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:36.036 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:36.037 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:36.286 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:36.287 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:36.287 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:36.287 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:36.288 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:36.288 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:36.288 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:36.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:36.288 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:36.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:36.289 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:36.289 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:36.289 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:36.290 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:36.290 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:36.290 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:36.290 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:36.291 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:36.291 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:36.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:36.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:36.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:36.292 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:36.292 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:36.292 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:36.293 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:36.293 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:36.293 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:36.293 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:36.294 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:36.294 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:36.294 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:36.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:36.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:36.295 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:36.295 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:36.295 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:36.296 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:36.296 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:36.296 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:36.296 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:36.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:36.297 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:36.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:36.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:36.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:36.298 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:36.298 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:36.298 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:36.298 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:36.299 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:36.299 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:36.299 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:36.299 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:36.300 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:36.300 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:36.300 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:36.300 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:36.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:36.301 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:36.301 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:36.301 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:36.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:36.302 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:36.303 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:41.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:41.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:41.306 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:41.629 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:41.644 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:41.644 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:41.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:41.644 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:41.645 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:41.645 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:41.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:41.645 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:41.647 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:46.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:46.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:46.649 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:46.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:46.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:46.965 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:46.966 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:46.966 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:46.966 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:46.966 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:46.967 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:46.967 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:46.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:46.967 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:46.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:46.968 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:46.968 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:46.968 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:46.968 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:46.969 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:46.969 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:46.969 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:46.969 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:46.969 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:46.970 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:46.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:46.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:46.970 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:46.970 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:46.971 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:46.971 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:46.971 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:46.971 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:46.972 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:46.972 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:46.972 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:46.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:46.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:46.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:46.973 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:46.973 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:46.973 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:46.973 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:46.974 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:46.974 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:46.974 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:46.974 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:46.974 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:46.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:46.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:46.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:46.975 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:46.975 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:46.976 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:46.976 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:46.976 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:46.976 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:46.976 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:46.977 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:46.977 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:46.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:46.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:46.977 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:46.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:46.978 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:46.978 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:46.978 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:46.978 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:46.979 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:46.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:51.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:52.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:52.053 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:52.095 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:52.096 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:52.380 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:52.380 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:52.381 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:52.381 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:52.382 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:52.382 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:52.382 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:52.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:52.383 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:52.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:52.384 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:52.384 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:52.385 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:52.385 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:52.386 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:52.386 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:52.386 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:52.387 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:52.387 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:52.387 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:52.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:52.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:52.388 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:52.388 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:52.388 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:52.388 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:52.388 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:52.389 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:52.389 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:52.389 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:52.389 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:52.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:52.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:52.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:52.390 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:52.390 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:52.391 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:52.391 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:52.391 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:52.391 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:52.392 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:52.392 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:52.392 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:52.392 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:52.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:52.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:52.393 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:52.393 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:52.393 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:52.393 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:52.394 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:52.394 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:52.394 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:52.394 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:52.394 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:52.395 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:52.395 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:52.395 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:52.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:52.396 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:52.396 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:52.396 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:52.396 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:52.396 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:52.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:13:57.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:57.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:13:57.440 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:13:57.484 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:13:57.484 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:13:57.759 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:13:57.760 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:13:57.761 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:13:57.761 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:13:57.761 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:13:57.762 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:13:57.762 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:57.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:57.763 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:13:57.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:57.764 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:13:57.765 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:57.765 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:57.765 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:13:57.766 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:57.766 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:13:57.766 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:57.767 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:13:57.767 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:13:57.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:57.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:57.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:57.768 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:13:57.768 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:57.769 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:57.769 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:13:57.769 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:57.770 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:57.770 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:57.770 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:13:57.771 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:57.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:57.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:57.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:57.772 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:13:57.772 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:57.773 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:57.773 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:57.773 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:57.774 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:57.774 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:57.774 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:57.775 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:57.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:57.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:57.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:13:57.776 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:13:57.776 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:13:57.777 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:13:57.777 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:13:57.777 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:13:57.778 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:13:57.778 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:13:57.778 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:13:57.778 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:13:57.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:57.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:57.779 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:13:57.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:13:57.780 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:13:57.781 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:13:57.781 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:13:57.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:13:57.781 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:13:57.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:14:02.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:03.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:03.390 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:03.434 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:03.434 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:03.699 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:03.700 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:03.700 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:03.700 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:03.700 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:03.701 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:14:03.701 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:03.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:03.701 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:14:03.703 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:03.703 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:14:03.703 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:03.703 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:14:03.704 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:03.706 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:03.706 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:14:03.706 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:03.706 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:14:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:14:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:14:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:03.714 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:03.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:14:08.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:08.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:08.718 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:08.760 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:08.760 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:09.035 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:14:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:14:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:14:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:14:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:09.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:09.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:09.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:09.040 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:09.041 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:09.041 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:14:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:14:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:14:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:14:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:09.051 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:09.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:14:14.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:14.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:14.055 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:14.104 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:14.105 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:14.393 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:14.393 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:14.393 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:14.393 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:14.394 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:14.394 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:14:14.394 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:14.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:14.394 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:14.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:14.395 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:14.395 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:14.395 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:14.395 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755280.842086_522.1", 2020-11-19T03:14:14.396 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:14.396 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:14:14.396 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:14.396 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:08:00.842086Z", 2020-11-19T03:14:14.396 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:14.396 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:14.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:14.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:14.397 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:14.397 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:14.397 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:14.398 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:14:14.398 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:14.398 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:14.398 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:14.398 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:14:14.399 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:14.399 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:14.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:14.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:14.399 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:14.399 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:14.400 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:14.400 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:14.400 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:14.400 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:14.400 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:14.401 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:14.401 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:14.401 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:14.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:14.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:14.402 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:14.402 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:14.402 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:14.402 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:14.402 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:14.402 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:14.403 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:14.403 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:14.403 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:14.403 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:14.403 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:14:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:14:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:14.405 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:14.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755280.842086_522.1 2020-11-19T03:14:19.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:19.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:19.411 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:19.455 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:19.456 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:19.733 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:19.734 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:19.734 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:19.734 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:19.734 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:19.734 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:14:19.735 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:19.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:19.735 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:19.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:19.735 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:19.736 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:19.736 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:19.736 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755551.492478_683.1", 2020-11-19T03:14:19.736 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:19.736 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:14:19.737 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:19.737 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:12:31.492478Z", 2020-11-19T03:14:19.737 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:19.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:19.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:19.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:19.738 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:19.738 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:19.738 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:19.739 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:14:19.739 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:19.739 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:19.739 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:19.739 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:14:19.740 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:19.740 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:19.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:19.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:19.740 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:19.741 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:19.741 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:19.741 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:19.741 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:19.741 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:19.742 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:19.742 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:19.742 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:19.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:19.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:19.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:19.743 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:19.743 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:19.743 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:19.744 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:19.744 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:19.744 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:19.744 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:19.745 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:19.745 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:19.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:19.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:19.745 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:19.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:19.746 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:19.746 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:14:19.746 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:14:19.746 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:19.747 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:19.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755551.492478_683.1 2020-11-19T03:14:24.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:24.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:24.750 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:24.795 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:24.795 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:25.063 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:25.063 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:25.063 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:25.063 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:25.064 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:25.064 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:14:25.064 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:25.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:25.065 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:25.066 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755551.492478_683.1", 2020-11-19T03:14:25.066 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:25.066 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:14:25.066 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:25.066 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:12:31.492478Z", 2020-11-19T03:14:25.067 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:25.067 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:25.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:25.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:25.068 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:25.068 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:25.068 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:25.068 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:14:25.068 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:14:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:25.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:25.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:25.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:25.070 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:25.070 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:25.070 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:25.071 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:25.071 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:25.071 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:25.071 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:25.071 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:25.072 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:25.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:25.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:25.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:25.073 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:25.073 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:25.073 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:25.073 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:25.073 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:25.074 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:25.074 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:25.074 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:25.074 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:25.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:25.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:25.075 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:25.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:25.075 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:25.076 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:14:25.076 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:14:25.076 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:25.076 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:25.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755551.492478_683.1 2020-11-19T03:14:30.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:30.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:30.421 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:30.465 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:30.465 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:30.734 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:30.734 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:30.734 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:30.735 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:30.735 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:30.735 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:14:30.735 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:30.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:30.736 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:30.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:30.736 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:30.736 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:30.737 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:30.737 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755551.492478_683.1", 2020-11-19T03:14:30.737 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:30.737 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:14:30.737 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:30.738 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:12:31.492478Z", 2020-11-19T03:14:30.738 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:30.738 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:30.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:30.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:30.739 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:30.739 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:30.739 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:30.739 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:14:30.740 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:30.740 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:30.740 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:30.740 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:14:30.740 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:30.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:30.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:30.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:30.741 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:30.741 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:30.742 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:30.742 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:30.742 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:30.742 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:30.742 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:30.743 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:30.743 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:30.743 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:30.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:30.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:30.744 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:30.744 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:30.744 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:30.744 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:30.744 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:30.745 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:30.745 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:30.745 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:30.745 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:30.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:30.747 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:30.747 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:30.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:30.748 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:30.748 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:14:30.748 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:14:30.748 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:30.749 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:30.750 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755555.445286_715.1 target=1_1605755551.492478_683.1 2020-11-19T03:14:35.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:35.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:35.752 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:35.792 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:35.792 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:36.066 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:36.067 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:36.067 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:36.067 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:36.067 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:36.067 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:14:36.068 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:36.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.068 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:36.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:36.069 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:36.069 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:36.069 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:36.069 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755555.445286_715.1", 2020-11-19T03:14:36.069 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:36.070 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:14:36.070 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:36.070 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:12:35.445286Z", 2020-11-19T03:14:36.070 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:36.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:36.071 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:36.071 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:36.072 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:36.072 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:14:36.072 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:36.072 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:36.072 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:36.073 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:14:36.073 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:36.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:14:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:14:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.080 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:36.081 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:14:36.082 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:14:36.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:36.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:36.083 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:36.123 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:36.123 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:36.371 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:14:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:14:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:14:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:14:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:14:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:14:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:14:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:14:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:14:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:14:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:14:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755555.445286_715.1", 2020-11-19T03:14:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:14:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:14:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:14:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:12:35.445286Z", 2020-11-19T03:14:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:14:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:14:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:14:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:14:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:14:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:14:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:14:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:14:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:14:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:14:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:14:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:14:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:14:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:14:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:14:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:14:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:14:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:14:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:14:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:14:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:14:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:14:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:14:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:14:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:14:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:14:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:14:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:14:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:14:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:14:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:14:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:14:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:36.385 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:14:36.386 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:14:36.386 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:14:36.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:36.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:36.388 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:36.428 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:36.429 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:36.796 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:36.796 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:36.797 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:36.797 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:36.797 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:36.797 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:14:36.798 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:36.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.798 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:36.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:36.798 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:36.799 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:36.799 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:36.799 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755555.445286_715.1", 2020-11-19T03:14:36.799 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:36.799 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:14:36.800 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:36.800 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:12:35.445286Z", 2020-11-19T03:14:36.800 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:14:36.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:36.801 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:36.801 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:36.801 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:36.802 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:14:36.802 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:36.802 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:36.802 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:36.802 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:14:36.803 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:36.803 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:36.803 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:36.804 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:36.804 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:36.804 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:36.804 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:36.804 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:36.805 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:36.805 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:36.805 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:36.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:36.806 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:36.806 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:36.806 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:14:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:14:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:14:36.808 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:14:36.808 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.808 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.808 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:36.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:36.809 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:36.809 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:14:36.809 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:14:36.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:36.810 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:36.811 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:14:36.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'atsuux-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:14:36.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'atsuux-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:14:36.812 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket atsuux-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:14:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:36.855 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:37.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'atsuux-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:14:37.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'atsuux-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:14:37.221 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket atsuux-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:14:37.301 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:37.301 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:37.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'atsuux-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:14:37.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'atsuux-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:14:37.668 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket atsuux-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:14:37.762 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:37.762 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:38.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:38.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:38.111 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:38.189 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:38.190 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:38.482 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:14:38.482 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000078", 2020-11-19T03:14:38.483 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.history", 2020-11-19T03:14:38.483 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.fdf730f5-4390-45ac-9d84-6887e17cb661.0", 2020-11-19T03:14:38.483 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000070", 2020-11-19T03:14:38.483 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000104", 2020-11-19T03:14:38.483 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24:8", 2020-11-19T03:14:38.484 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:3", 2020-11-19T03:14:38.484 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-49:d71482d6-15fd-4715-9992-a25958dcb538.5361.1:5", 2020-11-19T03:14:38.484 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000026", 2020-11-19T03:14:38.484 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:1", 2020-11-19T03:14:38.484 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:3", 2020-11-19T03:14:38.485 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000028", 2020-11-19T03:14:38.485 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.2", 2020-11-19T03:14:38.485 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.27-atsuux-27", 2020-11-19T03:14:38.485 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000040", 2020-11-19T03:14:38.486 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.39-atsuux-39", 2020-11-19T03:14:38.486 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.full-sync.index.0", 2020-11-19T03:14:38.486 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000015", 2020-11-19T03:14:38.486 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:7", 2020-11-19T03:14:38.486 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000069", 2020-11-19T03:14:38.487 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000095", 2020-11-19T03:14:38.487 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000003", 2020-11-19T03:14:38.487 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000047", 2020-11-19T03:14:38.487 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.36-atsuux-36", 2020-11-19T03:14:38.487 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.18-atsuux-18", 2020-11-19T03:14:38.488 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000035", 2020-11-19T03:14:38.488 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.3-atsuux-52", 2020-11-19T03:14:38.488 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22:8", 2020-11-19T03:14:38.488 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000037", 2020-11-19T03:14:38.488 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20:8", 2020-11-19T03:14:38.489 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.6-atsuux-55", 2020-11-19T03:14:38.489 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000024", 2020-11-19T03:14:38.489 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000000", 2020-11-19T03:14:38.489 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000031", 2020-11-19T03:14:38.490 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.15-atsuux-15", 2020-11-19T03:14:38.490 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.3", 2020-11-19T03:14:38.490 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.5-atsuux-48", 2020-11-19T03:14:38.490 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.5-atsuux-54", 2020-11-19T03:14:38.490 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:5", 2020-11-19T03:14:38.491 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:3", 2020-11-19T03:14:38.491 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:3", 2020-11-19T03:14:38.491 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:3", 2020-11-19T03:14:38.491 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000076", 2020-11-19T03:14:38.491 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000113", 2020-11-19T03:14:38.492 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000005", 2020-11-19T03:14:38.492 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.log.43cc2be5-65c1-4813-aee6-d404c9e85e90.0", 2020-11-19T03:14:38.492 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000011", 2020-11-19T03:14:38.492 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000058", 2020-11-19T03:14:38.493 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000052", 2020-11-19T03:14:38.493 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.12-atsuux-12", 2020-11-19T03:14:38.493 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.32-atsuux-32", 2020-11-19T03:14:38.493 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.11-atsuux-11", 2020-11-19T03:14:38.493 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.2", 2020-11-19T03:14:38.494 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000088", 2020-11-19T03:14:38.494 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000080", 2020-11-19T03:14:38.494 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000090", 2020-11-19T03:14:38.494 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13:1", 2020-11-19T03:14:38.495 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000110", 2020-11-19T03:14:38.495 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000096", 2020-11-19T03:14:38.495 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000087", 2020-11-19T03:14:38.495 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000008", 2020-11-19T03:14:38.495 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2-atsuux-43", 2020-11-19T03:14:38.495 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:8", 2020-11-19T03:14:38.496 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26:8", 2020-11-19T03:14:38.496 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36:10", 2020-11-19T03:14:38.496 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.5-atsuux-5", 2020-11-19T03:14:38.497 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000006", 2020-11-19T03:14:38.497 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000029", 2020-11-19T03:14:38.497 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000089", 2020-11-19T03:14:38.497 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000022", 2020-11-19T03:14:38.497 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.1.tLJJo3DHj8V.0", 2020-11-19T03:14:38.498 INFO:teuthology.orchestra.run.smithi086.stdout: "bilog.trim", 2020-11-19T03:14:38.498 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000016", 2020-11-19T03:14:38.498 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:1", 2020-11-19T03:14:38.498 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000041", 2020-11-19T03:14:38.499 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:3", 2020-11-19T03:14:38.499 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000018", 2020-11-19T03:14:38.499 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000092", 2020-11-19T03:14:38.499 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37:10", 2020-11-19T03:14:38.499 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.30-atsuux-30", 2020-11-19T03:14:38.500 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.fdf730f5-4390-45ac-9d84-6887e17cb661.1", 2020-11-19T03:14:38.500 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.29-atsuux-29", 2020-11-19T03:14:38.500 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000014", 2020-11-19T03:14:38.500 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5361.1-atsuux-49", 2020-11-19T03:14:38.501 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000112", 2020-11-19T03:14:38.501 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000007", 2020-11-19T03:14:38.501 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000021", 2020-11-19T03:14:38.501 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24:8", 2020-11-19T03:14:38.502 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000064", 2020-11-19T03:14:38.502 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000071", 2020-11-19T03:14:38.502 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000074", 2020-11-19T03:14:38.502 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.2.iN5AqTbS0za.0", 2020-11-19T03:14:38.503 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21:8", 2020-11-19T03:14:38.503 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000081", 2020-11-19T03:14:38.503 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.13-atsuux-13", 2020-11-19T03:14:38.503 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000009", 2020-11-19T03:14:38.504 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000121", 2020-11-19T03:14:38.504 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.1", 2020-11-19T03:14:38.504 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.9-atsuux-58", 2020-11-19T03:14:38.504 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:7", 2020-11-19T03:14:38.504 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000125", 2020-11-19T03:14:38.505 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000082", 2020-11-19T03:14:38.505 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.2-atsuux-51", 2020-11-19T03:14:38.505 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.full-sync.index.1", 2020-11-19T03:14:38.505 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-48:d71482d6-15fd-4715-9992-a25958dcb538.5149.5:8", 2020-11-19T03:14:38.506 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000105", 2020-11-19T03:14:38.506 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000059", 2020-11-19T03:14:38.506 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.fdf730f5-4390-45ac-9d84-6887e17cb661.3", 2020-11-19T03:14:38.506 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000077", 2020-11-19T03:14:38.506 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000032", 2020-11-19T03:14:38.507 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1-atsuux-42", 2020-11-19T03:14:38.507 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-48:d71482d6-15fd-4715-9992-a25958dcb538.5149.5:8", 2020-11-19T03:14:38.507 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.34-atsuux-34", 2020-11-19T03:14:38.507 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.fdf730f5-4390-45ac-9d84-6887e17cb661.1", 2020-11-19T03:14:38.507 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.9-atsuux-9", 2020-11-19T03:14:38.508 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15:1", 2020-11-19T03:14:38.508 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.14-atsuux-14", 2020-11-19T03:14:38.508 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000053", 2020-11-19T03:14:38.508 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000091", 2020-11-19T03:14:38.508 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000065", 2020-11-19T03:14:38.509 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:8", 2020-11-19T03:14:38.509 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000083", 2020-11-19T03:14:38.509 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000010", 2020-11-19T03:14:38.509 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:5", 2020-11-19T03:14:38.509 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.2-atsuux-2", 2020-11-19T03:14:38.510 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.1-atsuux-1", 2020-11-19T03:14:38.510 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38:10", 2020-11-19T03:14:38.510 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.8-atsuux-8", 2020-11-19T03:14:38.510 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:8", 2020-11-19T03:14:38.510 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.7-atsuux-56", 2020-11-19T03:14:38.511 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status.shard.2", 2020-11-19T03:14:38.511 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.33-atsuux-33", 2020-11-19T03:14:38.511 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000045", 2020-11-19T03:14:38.511 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:5", 2020-11-19T03:14:38.511 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000002", 2020-11-19T03:14:38.512 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23:8", 2020-11-19T03:14:38.512 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000116", 2020-11-19T03:14:38.512 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-46:d71482d6-15fd-4715-9992-a25958dcb538.5149.3:8", 2020-11-19T03:14:38.512 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13:1", 2020-11-19T03:14:38.512 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000034", 2020-11-19T03:14:38.513 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.41-atsuux-41", 2020-11-19T03:14:38.513 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000101", 2020-11-19T03:14:38.513 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25:8", 2020-11-19T03:14:38.513 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-49:d71482d6-15fd-4715-9992-a25958dcb538.5361.1:5", 2020-11-19T03:14:38.513 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status.shard.1", 2020-11-19T03:14:38.514 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.22-atsuux-22", 2020-11-19T03:14:38.514 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:8", 2020-11-19T03:14:38.514 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000079", 2020-11-19T03:14:38.514 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000049", 2020-11-19T03:14:38.514 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000117", 2020-11-19T03:14:38.515 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000044", 2020-11-19T03:14:38.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20:8", 2020-11-19T03:14:38.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:7", 2020-11-19T03:14:38.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:1", 2020-11-19T03:14:38.516 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000066", 2020-11-19T03:14:38.516 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.0", 2020-11-19T03:14:38.516 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000068", 2020-11-19T03:14:38.516 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000085", 2020-11-19T03:14:38.516 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.31-atsuux-31", 2020-11-19T03:14:38.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:1", 2020-11-19T03:14:38.517 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000073", 2020-11-19T03:14:38.517 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:14:38.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:3", 2020-11-19T03:14:38.517 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000038", 2020-11-19T03:14:38.518 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.28-atsuux-28", 2020-11-19T03:14:38.518 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.1-atsuux-44", 2020-11-19T03:14:38.518 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.full-sync.index.3", 2020-11-19T03:14:38.518 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000118", 2020-11-19T03:14:38.518 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.fdf730f5-4390-45ac-9d84-6887e17cb661.3", 2020-11-19T03:14:38.519 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000036", 2020-11-19T03:14:38.519 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.fdf730f5-4390-45ac-9d84-6887e17cb661.2", 2020-11-19T03:14:38.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-47:d71482d6-15fd-4715-9992-a25958dcb538.5149.4:8", 2020-11-19T03:14:38.519 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.17-atsuux-17", 2020-11-19T03:14:38.519 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000103", 2020-11-19T03:14:38.520 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status.shard.3", 2020-11-19T03:14:38.520 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000119", 2020-11-19T03:14:38.520 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000098", 2020-11-19T03:14:38.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-47:d71482d6-15fd-4715-9992-a25958dcb538.5149.4:8", 2020-11-19T03:14:38.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:1", 2020-11-19T03:14:38.521 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status.shard.0", 2020-11-19T03:14:38.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:3", 2020-11-19T03:14:38.521 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.23-atsuux-23", 2020-11-19T03:14:38.521 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000027", 2020-11-19T03:14:38.521 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status", 2020-11-19T03:14:38.522 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000019", 2020-11-19T03:14:38.522 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000039", 2020-11-19T03:14:38.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:1", 2020-11-19T03:14:38.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:1", 2020-11-19T03:14:38.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14:1", 2020-11-19T03:14:38.523 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000100", 2020-11-19T03:14:38.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:1", 2020-11-19T03:14:38.523 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.38-atsuux-38", 2020-11-19T03:14:38.523 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000093", 2020-11-19T03:14:38.524 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000004", 2020-11-19T03:14:38.524 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000063", 2020-11-19T03:14:38.524 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000122", 2020-11-19T03:14:38.524 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.8-atsuux-57", 2020-11-19T03:14:38.524 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.10-atsuux-10", 2020-11-19T03:14:38.525 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000057", 2020-11-19T03:14:38.525 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000054", 2020-11-19T03:14:38.525 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000114", 2020-11-19T03:14:38.525 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.0", 2020-11-19T03:14:38.525 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.1", 2020-11-19T03:14:38.526 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000012", 2020-11-19T03:14:38.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37:10", 2020-11-19T03:14:38.526 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.4-atsuux-4", 2020-11-19T03:14:38.526 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.21-atsuux-21", 2020-11-19T03:14:38.526 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000084", 2020-11-19T03:14:38.527 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000043", 2020-11-19T03:14:38.527 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.40-atsuux-40", 2020-11-19T03:14:38.527 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.1-atsuux-50", 2020-11-19T03:14:38.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:3", 2020-11-19T03:14:38.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:3", 2020-11-19T03:14:38.528 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:5", 2020-11-19T03:14:38.528 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000111", 2020-11-19T03:14:38.528 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000048", 2020-11-19T03:14:38.528 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000020", 2020-11-19T03:14:38.528 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.7-atsuux-7", 2020-11-19T03:14:38.529 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.20-atsuux-20", 2020-11-19T03:14:38.529 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000099", 2020-11-19T03:14:38.529 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000056", 2020-11-19T03:14:38.529 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000072", 2020-11-19T03:14:38.529 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14:1", 2020-11-19T03:14:38.530 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.4-atsuux-53", 2020-11-19T03:14:38.530 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.0", 2020-11-19T03:14:38.530 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000062", 2020-11-19T03:14:38.530 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000115", 2020-11-19T03:14:38.530 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:1", 2020-11-19T03:14:38.531 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000033", 2020-11-19T03:14:38.531 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:5", 2020-11-19T03:14:38.531 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:5", 2020-11-19T03:14:38.531 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.25-atsuux-25", 2020-11-19T03:14:38.531 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000025", 2020-11-19T03:14:38.532 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000123", 2020-11-19T03:14:38.532 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000108", 2020-11-19T03:14:38.532 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000094", 2020-11-19T03:14:38.532 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.1", 2020-11-19T03:14:38.532 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000060", 2020-11-19T03:14:38.533 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000109", 2020-11-19T03:14:38.533 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25:8", 2020-11-19T03:14:38.533 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000013", 2020-11-19T03:14:38.533 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:8", 2020-11-19T03:14:38.533 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21:8", 2020-11-19T03:14:38.534 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15:1", 2020-11-19T03:14:38.534 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26:8", 2020-11-19T03:14:38.534 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000051", 2020-11-19T03:14:38.534 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.3-atsuux-3", 2020-11-19T03:14:38.534 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000106", 2020-11-19T03:14:38.535 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.35-atsuux-35", 2020-11-19T03:14:38.535 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.19-atsuux-19", 2020-11-19T03:14:38.535 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000017", 2020-11-19T03:14:38.535 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:1", 2020-11-19T03:14:38.535 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.3", 2020-11-19T03:14:38.536 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38:10", 2020-11-19T03:14:38.536 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22:8", 2020-11-19T03:14:38.536 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000023", 2020-11-19T03:14:38.536 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.37-atsuux-37", 2020-11-19T03:14:38.536 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23:8", 2020-11-19T03:14:38.537 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:1", 2020-11-19T03:14:38.537 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000107", 2020-11-19T03:14:38.537 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000046", 2020-11-19T03:14:38.537 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.3.se5mKyk2xje.0", 2020-11-19T03:14:38.537 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.full-sync.index.2", 2020-11-19T03:14:38.538 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.0.zsGPB45rJNB.0", 2020-11-19T03:14:38.538 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.16-atsuux-16", 2020-11-19T03:14:38.538 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.2-atsuux-45", 2020-11-19T03:14:38.538 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.2", 2020-11-19T03:14:38.538 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:7", 2020-11-19T03:14:38.539 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000067", 2020-11-19T03:14:38.539 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:8", 2020-11-19T03:14:38.539 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000086", 2020-11-19T03:14:38.539 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.fdf730f5-4390-45ac-9d84-6887e17cb661.2", 2020-11-19T03:14:38.539 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.fdf730f5-4390-45ac-9d84-6887e17cb661.0", 2020-11-19T03:14:38.540 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.3-atsuux-46", 2020-11-19T03:14:38.540 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.6-atsuux-6", 2020-11-19T03:14:38.540 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:8", 2020-11-19T03:14:38.540 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.26-atsuux-26", 2020-11-19T03:14:38.540 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36:10", 2020-11-19T03:14:38.541 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.19-atsuux-19", 2020-11-19T03:14:38.541 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:5", 2020-11-19T03:14:38.541 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000050", 2020-11-19T03:14:38.541 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000102", 2020-11-19T03:14:38.541 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:7", 2020-11-19T03:14:38.542 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.4-atsuux-47", 2020-11-19T03:14:38.542 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:3", 2020-11-19T03:14:38.542 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000124", 2020-11-19T03:14:38.542 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-46:d71482d6-15fd-4715-9992-a25958dcb538.5149.3:8", 2020-11-19T03:14:38.542 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000001", 2020-11-19T03:14:38.543 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000126", 2020-11-19T03:14:38.543 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000055", 2020-11-19T03:14:38.543 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000061", 2020-11-19T03:14:38.543 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:5", 2020-11-19T03:14:38.543 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000097", 2020-11-19T03:14:38.544 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.24-atsuux-24", 2020-11-19T03:14:38.544 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.3", 2020-11-19T03:14:38.544 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000042", 2020-11-19T03:14:38.544 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:1", 2020-11-19T03:14:38.544 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000075", 2020-11-19T03:14:38.545 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:14:38.545 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000120", 2020-11-19T03:14:38.545 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000030", 2020-11-19T03:14:38.545 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:3", 2020-11-19T03:14:38.545 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:8", 2020-11-19T03:14:38.546 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:7" 2020-11-19T03:14:38.546 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:14:38.546 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T03:14:38.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:38.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:38.548 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:38.598 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:38.598 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:38.892 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:14:38.893 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000078", 2020-11-19T03:14:38.893 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.history", 2020-11-19T03:14:38.894 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:7", 2020-11-19T03:14:38.894 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.fdf730f5-4390-45ac-9d84-6887e17cb661.0", 2020-11-19T03:14:38.894 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000070", 2020-11-19T03:14:38.895 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000104", 2020-11-19T03:14:38.895 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24:8", 2020-11-19T03:14:38.896 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-49:d71482d6-15fd-4715-9992-a25958dcb538.5361.1:5", 2020-11-19T03:14:38.896 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000026", 2020-11-19T03:14:38.896 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:3", 2020-11-19T03:14:38.897 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:5", 2020-11-19T03:14:38.897 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.0.L2qqs33OIrI.0", 2020-11-19T03:14:38.897 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:1", 2020-11-19T03:14:38.897 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:3", 2020-11-19T03:14:38.898 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000028", 2020-11-19T03:14:38.898 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-48:d71482d6-15fd-4715-9992-a25958dcb538.5149.5:8", 2020-11-19T03:14:38.898 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000040", 2020-11-19T03:14:38.898 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.39-atsuux-39", 2020-11-19T03:14:38.899 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.full-sync.index.0", 2020-11-19T03:14:38.899 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000015", 2020-11-19T03:14:38.899 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:7", 2020-11-19T03:14:38.899 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000069", 2020-11-19T03:14:38.899 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000095", 2020-11-19T03:14:38.900 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000003", 2020-11-19T03:14:38.900 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000047", 2020-11-19T03:14:38.900 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.36-atsuux-36", 2020-11-19T03:14:38.900 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.18-atsuux-18", 2020-11-19T03:14:38.900 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000035", 2020-11-19T03:14:38.901 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22:8", 2020-11-19T03:14:38.901 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000037", 2020-11-19T03:14:38.901 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20:8", 2020-11-19T03:14:38.901 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000024", 2020-11-19T03:14:38.902 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-46:d71482d6-15fd-4715-9992-a25958dcb538.5149.3:8", 2020-11-19T03:14:38.902 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000000", 2020-11-19T03:14:38.902 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000031", 2020-11-19T03:14:38.903 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-47:d71482d6-15fd-4715-9992-a25958dcb538.5149.4:8", 2020-11-19T03:14:38.903 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.15-atsuux-15", 2020-11-19T03:14:38.903 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.3", 2020-11-19T03:14:38.903 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.5-atsuux-48", 2020-11-19T03:14:38.904 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25:8", 2020-11-19T03:14:38.904 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.5-atsuux-54", 2020-11-19T03:14:38.904 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:5", 2020-11-19T03:14:38.904 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:3", 2020-11-19T03:14:38.905 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000076", 2020-11-19T03:14:38.905 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000113", 2020-11-19T03:14:38.905 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000005", 2020-11-19T03:14:38.905 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.43cc2be5-65c1-4813-aee6-d404c9e85e90.0", 2020-11-19T03:14:38.905 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000011", 2020-11-19T03:14:38.906 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000058", 2020-11-19T03:14:38.906 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000052", 2020-11-19T03:14:38.906 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.12-atsuux-12", 2020-11-19T03:14:38.906 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.11-atsuux-11", 2020-11-19T03:14:38.907 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000088", 2020-11-19T03:14:38.907 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000080", 2020-11-19T03:14:38.907 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000090", 2020-11-19T03:14:38.907 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000110", 2020-11-19T03:14:38.908 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000096", 2020-11-19T03:14:38.908 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:3", 2020-11-19T03:14:38.908 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:3", 2020-11-19T03:14:38.908 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000087", 2020-11-19T03:14:38.909 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.2", 2020-11-19T03:14:38.909 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000008", 2020-11-19T03:14:38.909 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2-atsuux-43", 2020-11-19T03:14:38.909 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:14:38.909 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26:8", 2020-11-19T03:14:38.910 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36:10", 2020-11-19T03:14:38.910 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.5-atsuux-5", 2020-11-19T03:14:38.910 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000006", 2020-11-19T03:14:38.910 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000029", 2020-11-19T03:14:38.911 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000089", 2020-11-19T03:14:38.911 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000022", 2020-11-19T03:14:38.911 INFO:teuthology.orchestra.run.smithi160.stdout: "bilog.trim", 2020-11-19T03:14:38.911 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000016", 2020-11-19T03:14:38.912 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:1", 2020-11-19T03:14:38.912 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000041", 2020-11-19T03:14:38.912 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:3", 2020-11-19T03:14:38.912 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000018", 2020-11-19T03:14:38.913 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000092", 2020-11-19T03:14:38.913 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38:10", 2020-11-19T03:14:38.913 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.1", 2020-11-19T03:14:38.913 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.fdf730f5-4390-45ac-9d84-6887e17cb661.1", 2020-11-19T03:14:38.914 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000014", 2020-11-19T03:14:38.914 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5361.1-atsuux-49", 2020-11-19T03:14:38.914 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000112", 2020-11-19T03:14:38.914 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000007", 2020-11-19T03:14:38.914 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000021", 2020-11-19T03:14:38.915 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000064", 2020-11-19T03:14:38.915 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000071", 2020-11-19T03:14:38.915 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000074", 2020-11-19T03:14:38.915 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21:8", 2020-11-19T03:14:38.916 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000081", 2020-11-19T03:14:38.916 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.13-atsuux-13", 2020-11-19T03:14:38.916 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000009", 2020-11-19T03:14:38.916 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000121", 2020-11-19T03:14:38.917 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.1", 2020-11-19T03:14:38.917 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:7", 2020-11-19T03:14:38.917 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:1", 2020-11-19T03:14:38.917 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000125", 2020-11-19T03:14:38.917 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000082", 2020-11-19T03:14:38.918 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.2-atsuux-51", 2020-11-19T03:14:38.918 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.full-sync.index.1", 2020-11-19T03:14:38.918 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000105", 2020-11-19T03:14:38.919 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000059", 2020-11-19T03:14:38.919 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.fdf730f5-4390-45ac-9d84-6887e17cb661.3", 2020-11-19T03:14:38.919 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:3", 2020-11-19T03:14:38.919 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000077", 2020-11-19T03:14:38.919 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:8", 2020-11-19T03:14:38.920 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000032", 2020-11-19T03:14:38.920 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1-atsuux-42", 2020-11-19T03:14:38.920 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-48:d71482d6-15fd-4715-9992-a25958dcb538.5149.5:8", 2020-11-19T03:14:38.921 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.34-atsuux-34", 2020-11-19T03:14:38.922 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.fdf730f5-4390-45ac-9d84-6887e17cb661.1", 2020-11-19T03:14:38.922 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.9-atsuux-9", 2020-11-19T03:14:38.923 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15:1", 2020-11-19T03:14:38.923 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.14-atsuux-14", 2020-11-19T03:14:38.924 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000053", 2020-11-19T03:14:38.924 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000091", 2020-11-19T03:14:38.924 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000065", 2020-11-19T03:14:38.924 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000083", 2020-11-19T03:14:38.925 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000010", 2020-11-19T03:14:38.925 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20:8", 2020-11-19T03:14:38.925 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.2-atsuux-2", 2020-11-19T03:14:38.925 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.1-atsuux-1", 2020-11-19T03:14:38.926 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38:10", 2020-11-19T03:14:38.926 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.8-atsuux-8", 2020-11-19T03:14:38.926 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:8", 2020-11-19T03:14:38.926 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.2", 2020-11-19T03:14:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:5", 2020-11-19T03:14:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000045", 2020-11-19T03:14:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000002", 2020-11-19T03:14:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23:8", 2020-11-19T03:14:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000116", 2020-11-19T03:14:38.928 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13:1", 2020-11-19T03:14:38.928 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000034", 2020-11-19T03:14:38.928 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.41-atsuux-41", 2020-11-19T03:14:38.928 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000101", 2020-11-19T03:14:38.928 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25:8", 2020-11-19T03:14:38.929 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.1.sdU1CRvBlUl.0", 2020-11-19T03:14:38.929 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.1", 2020-11-19T03:14:38.929 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.22-atsuux-22", 2020-11-19T03:14:38.929 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:8", 2020-11-19T03:14:38.929 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.3", 2020-11-19T03:14:38.930 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-49:d71482d6-15fd-4715-9992-a25958dcb538.5361.1:5", 2020-11-19T03:14:38.930 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000079", 2020-11-19T03:14:38.930 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000049", 2020-11-19T03:14:38.930 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000117", 2020-11-19T03:14:38.931 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000044", 2020-11-19T03:14:38.931 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24:8", 2020-11-19T03:14:38.931 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000066", 2020-11-19T03:14:38.931 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22:8", 2020-11-19T03:14:38.932 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:3", 2020-11-19T03:14:38.932 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:8", 2020-11-19T03:14:38.932 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000068", 2020-11-19T03:14:38.932 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000085", 2020-11-19T03:14:38.932 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.31-atsuux-31", 2020-11-19T03:14:38.933 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:1", 2020-11-19T03:14:38.933 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000073", 2020-11-19T03:14:38.933 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661", 2020-11-19T03:14:38.933 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.1", 2020-11-19T03:14:38.933 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000038", 2020-11-19T03:14:38.934 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15:1", 2020-11-19T03:14:38.934 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.28-atsuux-28", 2020-11-19T03:14:38.934 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.1-atsuux-44", 2020-11-19T03:14:38.934 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.full-sync.index.3", 2020-11-19T03:14:38.934 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000118", 2020-11-19T03:14:38.935 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.fdf730f5-4390-45ac-9d84-6887e17cb661.3", 2020-11-19T03:14:38.935 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000036", 2020-11-19T03:14:38.935 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.fdf730f5-4390-45ac-9d84-6887e17cb661.2", 2020-11-19T03:14:38.935 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-47:d71482d6-15fd-4715-9992-a25958dcb538.5149.4:8", 2020-11-19T03:14:38.935 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.17-atsuux-17", 2020-11-19T03:14:38.936 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21:8", 2020-11-19T03:14:38.936 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000103", 2020-11-19T03:14:38.936 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.3", 2020-11-19T03:14:38.936 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000119", 2020-11-19T03:14:38.937 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36:10", 2020-11-19T03:14:38.937 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000098", 2020-11-19T03:14:38.937 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:8", 2020-11-19T03:14:38.937 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37:10", 2020-11-19T03:14:38.938 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.0", 2020-11-19T03:14:38.938 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:3", 2020-11-19T03:14:38.938 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.23-atsuux-23", 2020-11-19T03:14:38.938 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000027", 2020-11-19T03:14:38.938 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status", 2020-11-19T03:14:38.939 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000019", 2020-11-19T03:14:38.939 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:5", 2020-11-19T03:14:38.939 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000039", 2020-11-19T03:14:38.939 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:1", 2020-11-19T03:14:38.939 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:1", 2020-11-19T03:14:38.940 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14:1", 2020-11-19T03:14:38.940 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000100", 2020-11-19T03:14:38.940 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:7", 2020-11-19T03:14:38.940 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.38-atsuux-38", 2020-11-19T03:14:38.940 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000093", 2020-11-19T03:14:38.941 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000004", 2020-11-19T03:14:38.941 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000063", 2020-11-19T03:14:38.941 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000122", 2020-11-19T03:14:38.941 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.8-atsuux-57", 2020-11-19T03:14:38.941 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.10-atsuux-10", 2020-11-19T03:14:38.942 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:1", 2020-11-19T03:14:38.942 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000057", 2020-11-19T03:14:38.942 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:7", 2020-11-19T03:14:38.942 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000054", 2020-11-19T03:14:38.942 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000114", 2020-11-19T03:14:38.943 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.2.HjElIKckgbD.0", 2020-11-19T03:14:38.943 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.0", 2020-11-19T03:14:38.943 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000012", 2020-11-19T03:14:38.944 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:1", 2020-11-19T03:14:38.944 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37:10", 2020-11-19T03:14:38.944 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.4-atsuux-4", 2020-11-19T03:14:38.944 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.21-atsuux-21", 2020-11-19T03:14:38.945 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000084", 2020-11-19T03:14:38.945 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000043", 2020-11-19T03:14:38.945 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.40-atsuux-40", 2020-11-19T03:14:38.945 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:3", 2020-11-19T03:14:38.945 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14:1", 2020-11-19T03:14:38.946 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:5", 2020-11-19T03:14:38.946 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000111", 2020-11-19T03:14:38.946 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000048", 2020-11-19T03:14:38.946 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000020", 2020-11-19T03:14:38.946 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.7-atsuux-7", 2020-11-19T03:14:38.947 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.20-atsuux-20", 2020-11-19T03:14:38.947 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000099", 2020-11-19T03:14:38.947 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000056", 2020-11-19T03:14:38.947 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13:1", 2020-11-19T03:14:38.947 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000072", 2020-11-19T03:14:38.948 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000062", 2020-11-19T03:14:38.948 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.0", 2020-11-19T03:14:38.948 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000115", 2020-11-19T03:14:38.948 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000033", 2020-11-19T03:14:38.948 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:5", 2020-11-19T03:14:38.949 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.25-atsuux-25", 2020-11-19T03:14:38.949 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000025", 2020-11-19T03:14:38.949 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000123", 2020-11-19T03:14:38.949 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000108", 2020-11-19T03:14:38.950 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000094", 2020-11-19T03:14:38.950 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000060", 2020-11-19T03:14:38.950 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000109", 2020-11-19T03:14:38.950 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000013", 2020-11-19T03:14:38.951 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:3", 2020-11-19T03:14:38.951 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:5", 2020-11-19T03:14:38.951 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.3.16glIem2Bvt.0", 2020-11-19T03:14:38.951 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:8", 2020-11-19T03:14:38.951 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000051", 2020-11-19T03:14:38.952 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.3-atsuux-3", 2020-11-19T03:14:38.952 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000106", 2020-11-19T03:14:38.952 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.19-atsuux-19", 2020-11-19T03:14:38.952 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000017", 2020-11-19T03:14:38.952 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000023", 2020-11-19T03:14:38.953 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.37-atsuux-37", 2020-11-19T03:14:38.953 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:1", 2020-11-19T03:14:38.953 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000107", 2020-11-19T03:14:38.953 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000046", 2020-11-19T03:14:38.953 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23:8", 2020-11-19T03:14:38.954 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.full-sync.index.2", 2020-11-19T03:14:38.954 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.16-atsuux-16", 2020-11-19T03:14:38.954 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.2-atsuux-45", 2020-11-19T03:14:38.954 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.2", 2020-11-19T03:14:38.954 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000067", 2020-11-19T03:14:38.955 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:8", 2020-11-19T03:14:38.955 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000086", 2020-11-19T03:14:38.955 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.fdf730f5-4390-45ac-9d84-6887e17cb661.2", 2020-11-19T03:14:38.955 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.0", 2020-11-19T03:14:38.955 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.fdf730f5-4390-45ac-9d84-6887e17cb661.0", 2020-11-19T03:14:38.956 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.3-atsuux-46", 2020-11-19T03:14:38.956 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.6-atsuux-6", 2020-11-19T03:14:38.956 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:8", 2020-11-19T03:14:38.956 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.2", 2020-11-19T03:14:38.956 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:1", 2020-11-19T03:14:38.957 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.26-atsuux-26", 2020-11-19T03:14:38.957 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.19-atsuux-19", 2020-11-19T03:14:38.957 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:5", 2020-11-19T03:14:38.957 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000050", 2020-11-19T03:14:38.957 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000102", 2020-11-19T03:14:38.958 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.4-atsuux-47", 2020-11-19T03:14:38.958 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000124", 2020-11-19T03:14:38.958 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:1", 2020-11-19T03:14:38.958 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-46:d71482d6-15fd-4715-9992-a25958dcb538.5149.3:8", 2020-11-19T03:14:38.958 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000001", 2020-11-19T03:14:38.959 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000126", 2020-11-19T03:14:38.959 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000055", 2020-11-19T03:14:38.959 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000061", 2020-11-19T03:14:38.959 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000097", 2020-11-19T03:14:38.959 INFO:teuthology.orchestra.run.smithi160.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.24-atsuux-24", 2020-11-19T03:14:38.960 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000042", 2020-11-19T03:14:38.960 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26:8", 2020-11-19T03:14:38.960 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:1", 2020-11-19T03:14:38.960 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000075", 2020-11-19T03:14:38.960 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000120", 2020-11-19T03:14:38.961 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000030", 2020-11-19T03:14:38.961 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:3", 2020-11-19T03:14:38.961 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.3", 2020-11-19T03:14:38.961 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.fdf730f5-4390-45ac-9d84-6887e17cb661:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:7" 2020-11-19T03:14:38.961 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:14:38.962 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T03:14:38.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:38.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:38.964 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:39.002 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:39.002 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:39.322 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:14:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000078", 2020-11-19T03:14:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.history", 2020-11-19T03:14:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:7", 2020-11-19T03:14:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000070", 2020-11-19T03:14:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000104", 2020-11-19T03:14:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:3", 2020-11-19T03:14:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000026", 2020-11-19T03:14:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:3", 2020-11-19T03:14:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:5", 2020-11-19T03:14:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000028", 2020-11-19T03:14:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.2", 2020-11-19T03:14:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-48:d71482d6-15fd-4715-9992-a25958dcb538.5149.5:8", 2020-11-19T03:14:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000040", 2020-11-19T03:14:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.39-atsuux-39", 2020-11-19T03:14:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.full-sync.index.0", 2020-11-19T03:14:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000015", 2020-11-19T03:14:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000069", 2020-11-19T03:14:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000095", 2020-11-19T03:14:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000003", 2020-11-19T03:14:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000047", 2020-11-19T03:14:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.36-atsuux-36", 2020-11-19T03:14:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.18-atsuux-18", 2020-11-19T03:14:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000035", 2020-11-19T03:14:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.3-atsuux-52", 2020-11-19T03:14:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000037", 2020-11-19T03:14:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.6-atsuux-55", 2020-11-19T03:14:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000024", 2020-11-19T03:14:39.328 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-46:d71482d6-15fd-4715-9992-a25958dcb538.5149.3:8", 2020-11-19T03:14:39.328 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000000", 2020-11-19T03:14:39.328 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000031", 2020-11-19T03:14:39.328 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-47:d71482d6-15fd-4715-9992-a25958dcb538.5149.4:8", 2020-11-19T03:14:39.328 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.15-atsuux-15", 2020-11-19T03:14:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.3", 2020-11-19T03:14:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.5-atsuux-48", 2020-11-19T03:14:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25:8", 2020-11-19T03:14:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:3", 2020-11-19T03:14:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:3", 2020-11-19T03:14:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000076", 2020-11-19T03:14:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000113", 2020-11-19T03:14:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000005", 2020-11-19T03:14:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.log.43cc2be5-65c1-4813-aee6-d404c9e85e90.0", 2020-11-19T03:14:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000011", 2020-11-19T03:14:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000058", 2020-11-19T03:14:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000052", 2020-11-19T03:14:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.12-atsuux-12", 2020-11-19T03:14:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.32-atsuux-32", 2020-11-19T03:14:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.11-atsuux-11", 2020-11-19T03:14:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.2", 2020-11-19T03:14:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000088", 2020-11-19T03:14:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000080", 2020-11-19T03:14:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000090", 2020-11-19T03:14:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13:1", 2020-11-19T03:14:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000110", 2020-11-19T03:14:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000096", 2020-11-19T03:14:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:3", 2020-11-19T03:14:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:3", 2020-11-19T03:14:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000087", 2020-11-19T03:14:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.2", 2020-11-19T03:14:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000008", 2020-11-19T03:14:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.2-atsuux-43", 2020-11-19T03:14:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:14:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:8", 2020-11-19T03:14:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.5-atsuux-5", 2020-11-19T03:14:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000006", 2020-11-19T03:14:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000029", 2020-11-19T03:14:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000089", 2020-11-19T03:14:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000022", 2020-11-19T03:14:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "bilog.trim", 2020-11-19T03:14:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000016", 2020-11-19T03:14:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000041", 2020-11-19T03:14:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.log.b2c8afed-318c-484c-8dd4-daab7a757391.1", 2020-11-19T03:14:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000018", 2020-11-19T03:14:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000092", 2020-11-19T03:14:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.3.nxknPZLUMV8.0", 2020-11-19T03:14:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37:10", 2020-11-19T03:14:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38:10", 2020-11-19T03:14:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.1", 2020-11-19T03:14:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.29-atsuux-29", 2020-11-19T03:14:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000014", 2020-11-19T03:14:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5361.1-atsuux-49", 2020-11-19T03:14:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000112", 2020-11-19T03:14:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000007", 2020-11-19T03:14:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000021", 2020-11-19T03:14:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24:8", 2020-11-19T03:14:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.2.GUmB3KrFkLD.0", 2020-11-19T03:14:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000064", 2020-11-19T03:14:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000071", 2020-11-19T03:14:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000074", 2020-11-19T03:14:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000081", 2020-11-19T03:14:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.13-atsuux-13", 2020-11-19T03:14:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000009", 2020-11-19T03:14:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000121", 2020-11-19T03:14:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.1", 2020-11-19T03:14:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5486.9-atsuux-58", 2020-11-19T03:14:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:1", 2020-11-19T03:14:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000125", 2020-11-19T03:14:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000082", 2020-11-19T03:14:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.full-sync.index.1", 2020-11-19T03:14:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-48:d71482d6-15fd-4715-9992-a25958dcb538.5149.5:8", 2020-11-19T03:14:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.log.b2c8afed-318c-484c-8dd4-daab7a757391.0", 2020-11-19T03:14:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000105", 2020-11-19T03:14:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000059", 2020-11-19T03:14:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:3", 2020-11-19T03:14:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000077", 2020-11-19T03:14:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:8", 2020-11-19T03:14:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000032", 2020-11-19T03:14:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.5097.1-atsuux-42", 2020-11-19T03:14:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.9-atsuux-9", 2020-11-19T03:14:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.14-atsuux-14", 2020-11-19T03:14:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000053", 2020-11-19T03:14:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000091", 2020-11-19T03:14:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000065", 2020-11-19T03:14:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:8", 2020-11-19T03:14:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000083", 2020-11-19T03:14:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000010", 2020-11-19T03:14:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:5", 2020-11-19T03:14:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20:8", 2020-11-19T03:14:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.2-atsuux-2", 2020-11-19T03:14:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.1-atsuux-1", 2020-11-19T03:14:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.8-atsuux-8", 2020-11-19T03:14:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status.shard.2", 2020-11-19T03:14:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:5", 2020-11-19T03:14:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.log.0e78ea22-1fd1-48d9-9be5-2a207091b0ca.0", 2020-11-19T03:14:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000045", 2020-11-19T03:14:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:5", 2020-11-19T03:14:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000002", 2020-11-19T03:14:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000116", 2020-11-19T03:14:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-46:d71482d6-15fd-4715-9992-a25958dcb538.5149.3:8", 2020-11-19T03:14:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000034", 2020-11-19T03:14:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.41-atsuux-41", 2020-11-19T03:14:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000101", 2020-11-19T03:14:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-49:d71482d6-15fd-4715-9992-a25958dcb538.5361.1:5", 2020-11-19T03:14:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status.shard.1", 2020-11-19T03:14:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.22-atsuux-22", 2020-11-19T03:14:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.3", 2020-11-19T03:14:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-49:d71482d6-15fd-4715-9992-a25958dcb538.5361.1:5", 2020-11-19T03:14:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000079", 2020-11-19T03:14:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000049", 2020-11-19T03:14:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000117", 2020-11-19T03:14:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000044", 2020-11-19T03:14:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20:8", 2020-11-19T03:14:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:7", 2020-11-19T03:14:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:1", 2020-11-19T03:14:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24:8", 2020-11-19T03:14:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000066", 2020-11-19T03:14:39.353 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22:8", 2020-11-19T03:14:39.353 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.0", 2020-11-19T03:14:39.353 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:3", 2020-11-19T03:14:39.353 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:8", 2020-11-19T03:14:39.353 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000068", 2020-11-19T03:14:39.354 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000085", 2020-11-19T03:14:39.354 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000073", 2020-11-19T03:14:39.354 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.1", 2020-11-19T03:14:39.354 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:3", 2020-11-19T03:14:39.354 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000038", 2020-11-19T03:14:39.355 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15:1", 2020-11-19T03:14:39.355 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.1-atsuux-44", 2020-11-19T03:14:39.355 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.full-sync.index.3", 2020-11-19T03:14:39.355 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000118", 2020-11-19T03:14:39.355 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000036", 2020-11-19T03:14:39.356 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.17-atsuux-17", 2020-11-19T03:14:39.356 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21:8", 2020-11-19T03:14:39.356 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000103", 2020-11-19T03:14:39.356 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status.shard.3", 2020-11-19T03:14:39.356 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000119", 2020-11-19T03:14:39.357 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36:10", 2020-11-19T03:14:39.357 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000098", 2020-11-19T03:14:39.357 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:8", 2020-11-19T03:14:39.357 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-47:d71482d6-15fd-4715-9992-a25958dcb538.5149.4:8", 2020-11-19T03:14:39.357 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:1", 2020-11-19T03:14:39.358 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37:10", 2020-11-19T03:14:39.358 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status.shard.0", 2020-11-19T03:14:39.358 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.23-atsuux-23", 2020-11-19T03:14:39.358 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000027", 2020-11-19T03:14:39.358 INFO:teuthology.orchestra.run.smithi086.stdout: "mdlog.sync-status", 2020-11-19T03:14:39.359 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000019", 2020-11-19T03:14:39.359 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:5", 2020-11-19T03:14:39.359 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000039", 2020-11-19T03:14:39.359 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000100", 2020-11-19T03:14:39.359 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:1", 2020-11-19T03:14:39.360 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:7", 2020-11-19T03:14:39.360 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.38-atsuux-38", 2020-11-19T03:14:39.360 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000093", 2020-11-19T03:14:39.360 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000004", 2020-11-19T03:14:39.360 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000063", 2020-11-19T03:14:39.361 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000122", 2020-11-19T03:14:39.361 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.10-atsuux-10", 2020-11-19T03:14:39.361 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:1", 2020-11-19T03:14:39.361 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000057", 2020-11-19T03:14:39.361 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:7", 2020-11-19T03:14:39.362 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000054", 2020-11-19T03:14:39.362 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000114", 2020-11-19T03:14:39.362 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.0", 2020-11-19T03:14:39.362 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.1", 2020-11-19T03:14:39.362 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000012", 2020-11-19T03:14:39.363 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:1", 2020-11-19T03:14:39.363 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.4-atsuux-4", 2020-11-19T03:14:39.363 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.21-atsuux-21", 2020-11-19T03:14:39.363 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000084", 2020-11-19T03:14:39.363 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000043", 2020-11-19T03:14:39.364 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.40-atsuux-40", 2020-11-19T03:14:39.364 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:3", 2020-11-19T03:14:39.364 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14:1", 2020-11-19T03:14:39.364 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000111", 2020-11-19T03:14:39.364 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000048", 2020-11-19T03:14:39.365 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000020", 2020-11-19T03:14:39.365 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.7-atsuux-7", 2020-11-19T03:14:39.365 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.20-atsuux-20", 2020-11-19T03:14:39.365 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000099", 2020-11-19T03:14:39.365 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000056", 2020-11-19T03:14:39.366 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13:1", 2020-11-19T03:14:39.366 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000072", 2020-11-19T03:14:39.366 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14:1", 2020-11-19T03:14:39.366 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.0", 2020-11-19T03:14:39.366 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000062", 2020-11-19T03:14:39.367 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.0", 2020-11-19T03:14:39.367 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000115", 2020-11-19T03:14:39.367 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:1", 2020-11-19T03:14:39.367 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000033", 2020-11-19T03:14:39.367 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:5", 2020-11-19T03:14:39.368 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.25-atsuux-25", 2020-11-19T03:14:39.368 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000025", 2020-11-19T03:14:39.368 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000123", 2020-11-19T03:14:39.369 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.0.+a4dc3tDszk.0", 2020-11-19T03:14:39.369 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000108", 2020-11-19T03:14:39.369 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000094", 2020-11-19T03:14:39.369 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.1", 2020-11-19T03:14:39.369 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000060", 2020-11-19T03:14:39.370 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000109", 2020-11-19T03:14:39.370 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25:8", 2020-11-19T03:14:39.370 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000013", 2020-11-19T03:14:39.370 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.1.6Ly6bXPxcR0.0", 2020-11-19T03:14:39.370 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:3", 2020-11-19T03:14:39.371 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:8", 2020-11-19T03:14:39.371 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:5", 2020-11-19T03:14:39.371 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21:8", 2020-11-19T03:14:39.371 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15:1", 2020-11-19T03:14:39.371 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26:8", 2020-11-19T03:14:39.372 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:8", 2020-11-19T03:14:39.372 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000051", 2020-11-19T03:14:39.372 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.3-atsuux-3", 2020-11-19T03:14:39.372 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000106", 2020-11-19T03:14:39.372 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.35-atsuux-35", 2020-11-19T03:14:39.373 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.19-atsuux-19", 2020-11-19T03:14:39.373 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000017", 2020-11-19T03:14:39.373 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:1", 2020-11-19T03:14:39.373 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.3", 2020-11-19T03:14:39.373 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38:10", 2020-11-19T03:14:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22:8", 2020-11-19T03:14:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000023", 2020-11-19T03:14:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.37-atsuux-37", 2020-11-19T03:14:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:1", 2020-11-19T03:14:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23:8", 2020-11-19T03:14:39.375 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:1", 2020-11-19T03:14:39.375 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000107", 2020-11-19T03:14:39.375 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000046", 2020-11-19T03:14:39.375 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23:8", 2020-11-19T03:14:39.375 INFO:teuthology.orchestra.run.smithi086.stdout: "meta.full-sync.index.2", 2020-11-19T03:14:39.376 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.16-atsuux-16", 2020-11-19T03:14:39.376 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.2-atsuux-45", 2020-11-19T03:14:39.376 INFO:teuthology.orchestra.run.smithi086.stdout: "data_log.2", 2020-11-19T03:14:39.376 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:7", 2020-11-19T03:14:39.376 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000067", 2020-11-19T03:14:39.377 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000086", 2020-11-19T03:14:39.377 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.0", 2020-11-19T03:14:39.377 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.3-atsuux-46", 2020-11-19T03:14:39.377 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.6-atsuux-6", 2020-11-19T03:14:39.377 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.2", 2020-11-19T03:14:39.378 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:1", 2020-11-19T03:14:39.378 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.26-atsuux-26", 2020-11-19T03:14:39.378 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36:10", 2020-11-19T03:14:39.378 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-02-d71482d6-15fd-4715-9992-a25958dcb538.4341.19-atsuux-19", 2020-11-19T03:14:39.378 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000050", 2020-11-19T03:14:39.379 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000102", 2020-11-19T03:14:39.379 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:7", 2020-11-19T03:14:39.379 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.5149.4-atsuux-47", 2020-11-19T03:14:39.379 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:3", 2020-11-19T03:14:39.379 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000124", 2020-11-19T03:14:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:1", 2020-11-19T03:14:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000001", 2020-11-19T03:14:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000126", 2020-11-19T03:14:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000055", 2020-11-19T03:14:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000061", 2020-11-19T03:14:39.381 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:5", 2020-11-19T03:14:39.381 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000097", 2020-11-19T03:14:39.381 INFO:teuthology.orchestra.run.smithi086.stdout: "2020-11-19-03-d71482d6-15fd-4715-9992-a25958dcb538.4341.24-atsuux-24", 2020-11-19T03:14:39.381 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c.3", 2020-11-19T03:14:39.381 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000042", 2020-11-19T03:14:39.382 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26:8", 2020-11-19T03:14:39.382 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000075", 2020-11-19T03:14:39.382 INFO:teuthology.orchestra.run.smithi086.stdout: "datalog.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c", 2020-11-19T03:14:39.382 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000120", 2020-11-19T03:14:39.382 INFO:teuthology.orchestra.run.smithi086.stdout: "obj_delete_at_hint.0000000030", 2020-11-19T03:14:39.383 INFO:teuthology.orchestra.run.smithi086.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.3", 2020-11-19T03:14:39.383 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket.sync-status.c9a90d7d-dd43-4d9d-9b12-b4d640ca300c:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:8" 2020-11-19T03:14:39.383 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:14:39.383 INFO:teuthology.orchestra.run.smithi086.stdout: 2020-11-19T03:14:39.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:39.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:39.385 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:39.437 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:39.438 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:39.689 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:14:40.342 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000078", 2020-11-19T03:14:40.342 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.history", 2020-11-19T03:14:40.342 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:7", 2020-11-19T03:14:40.342 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000070", 2020-11-19T03:14:40.343 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000104", 2020-11-19T03:14:40.343 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000026", 2020-11-19T03:14:40.343 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:3", 2020-11-19T03:14:40.343 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:5", 2020-11-19T03:14:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000028", 2020-11-19T03:14:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-48:d71482d6-15fd-4715-9992-a25958dcb538.5149.5:8", 2020-11-19T03:14:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000040", 2020-11-19T03:14:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.full-sync.index.0", 2020-11-19T03:14:40.344 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.3.ruZPxJCpVan.0", 2020-11-19T03:14:40.345 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000015", 2020-11-19T03:14:40.345 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000069", 2020-11-19T03:14:40.345 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000095", 2020-11-19T03:14:40.345 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000003", 2020-11-19T03:14:40.345 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000047", 2020-11-19T03:14:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000035", 2020-11-19T03:14:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000037", 2020-11-19T03:14:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000024", 2020-11-19T03:14:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-46:d71482d6-15fd-4715-9992-a25958dcb538.5149.3:8", 2020-11-19T03:14:40.346 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000000", 2020-11-19T03:14:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000031", 2020-11-19T03:14:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-47:d71482d6-15fd-4715-9992-a25958dcb538.5149.4:8", 2020-11-19T03:14:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.3", 2020-11-19T03:14:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-25:d71482d6-15fd-4715-9992-a25958dcb538.4341.25:8", 2020-11-19T03:14:40.347 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000076", 2020-11-19T03:14:40.348 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000113", 2020-11-19T03:14:40.349 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000005", 2020-11-19T03:14:40.349 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.43cc2be5-65c1-4813-aee6-d404c9e85e90.0", 2020-11-19T03:14:40.349 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000011", 2020-11-19T03:14:40.350 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000058", 2020-11-19T03:14:40.350 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000052", 2020-11-19T03:14:40.350 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000088", 2020-11-19T03:14:40.350 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000080", 2020-11-19T03:14:40.350 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000090", 2020-11-19T03:14:40.351 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000110", 2020-11-19T03:14:40.351 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000096", 2020-11-19T03:14:40.351 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:3", 2020-11-19T03:14:40.351 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:3", 2020-11-19T03:14:40.351 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000087", 2020-11-19T03:14:40.352 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.2", 2020-11-19T03:14:40.352 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000008", 2020-11-19T03:14:40.352 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.d71482d6-15fd-4715-9992-a25958dcb538", 2020-11-19T03:14:40.352 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000006", 2020-11-19T03:14:40.352 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000029", 2020-11-19T03:14:40.353 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000089", 2020-11-19T03:14:40.353 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000022", 2020-11-19T03:14:40.353 INFO:teuthology.orchestra.run.smithi160.stdout: "bilog.trim", 2020-11-19T03:14:40.353 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000016", 2020-11-19T03:14:40.354 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.0.hi3kX0135am.0", 2020-11-19T03:14:40.354 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000041", 2020-11-19T03:14:40.354 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.b2c8afed-318c-484c-8dd4-daab7a757391.1", 2020-11-19T03:14:40.354 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000018", 2020-11-19T03:14:40.354 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000092", 2020-11-19T03:14:40.355 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-38:d71482d6-15fd-4715-9992-a25958dcb538.4341.38:10", 2020-11-19T03:14:40.355 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.1", 2020-11-19T03:14:40.355 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000014", 2020-11-19T03:14:40.355 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000112", 2020-11-19T03:14:40.355 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000007", 2020-11-19T03:14:40.355 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000021", 2020-11-19T03:14:40.356 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000064", 2020-11-19T03:14:40.356 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000071", 2020-11-19T03:14:40.356 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000074", 2020-11-19T03:14:40.356 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000081", 2020-11-19T03:14:40.356 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000009", 2020-11-19T03:14:40.357 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000121", 2020-11-19T03:14:40.357 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.1", 2020-11-19T03:14:40.357 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:1", 2020-11-19T03:14:40.357 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000125", 2020-11-19T03:14:40.357 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000082", 2020-11-19T03:14:40.358 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.full-sync.index.1", 2020-11-19T03:14:40.358 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.b2c8afed-318c-484c-8dd4-daab7a757391.0", 2020-11-19T03:14:40.358 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000105", 2020-11-19T03:14:40.358 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000059", 2020-11-19T03:14:40.358 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:3", 2020-11-19T03:14:40.359 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000077", 2020-11-19T03:14:40.359 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:8", 2020-11-19T03:14:40.359 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000032", 2020-11-19T03:14:40.359 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000053", 2020-11-19T03:14:40.359 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000091", 2020-11-19T03:14:40.360 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000065", 2020-11-19T03:14:40.360 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000083", 2020-11-19T03:14:40.360 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000010", 2020-11-19T03:14:40.360 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-20:d71482d6-15fd-4715-9992-a25958dcb538.4341.20:8", 2020-11-19T03:14:40.360 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.2", 2020-11-19T03:14:40.361 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:5", 2020-11-19T03:14:40.361 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.0e78ea22-1fd1-48d9-9be5-2a207091b0ca.0", 2020-11-19T03:14:40.361 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000045", 2020-11-19T03:14:40.361 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000002", 2020-11-19T03:14:40.361 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000116", 2020-11-19T03:14:40.362 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000034", 2020-11-19T03:14:40.362 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000101", 2020-11-19T03:14:40.362 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.1", 2020-11-19T03:14:40.362 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.3", 2020-11-19T03:14:40.362 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-49:d71482d6-15fd-4715-9992-a25958dcb538.5361.1:5", 2020-11-19T03:14:40.363 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000079", 2020-11-19T03:14:40.363 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000049", 2020-11-19T03:14:40.363 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000117", 2020-11-19T03:14:40.363 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000044", 2020-11-19T03:14:40.363 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-24:d71482d6-15fd-4715-9992-a25958dcb538.4341.24:8", 2020-11-19T03:14:40.364 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000066", 2020-11-19T03:14:40.364 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-22:d71482d6-15fd-4715-9992-a25958dcb538.4341.22:8", 2020-11-19T03:14:40.364 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:3", 2020-11-19T03:14:40.364 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:8", 2020-11-19T03:14:40.364 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000068", 2020-11-19T03:14:40.365 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000085", 2020-11-19T03:14:40.365 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000073", 2020-11-19T03:14:40.365 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.1", 2020-11-19T03:14:40.365 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000038", 2020-11-19T03:14:40.365 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-15:d71482d6-15fd-4715-9992-a25958dcb538.4341.15:1", 2020-11-19T03:14:40.366 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.full-sync.index.3", 2020-11-19T03:14:40.366 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000118", 2020-11-19T03:14:40.366 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000036", 2020-11-19T03:14:40.366 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-21:d71482d6-15fd-4715-9992-a25958dcb538.4341.21:8", 2020-11-19T03:14:40.366 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000103", 2020-11-19T03:14:40.367 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.3", 2020-11-19T03:14:40.367 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000119", 2020-11-19T03:14:40.367 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-36:d71482d6-15fd-4715-9992-a25958dcb538.4341.36:10", 2020-11-19T03:14:40.367 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000098", 2020-11-19T03:14:40.367 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:8", 2020-11-19T03:14:40.368 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-37:d71482d6-15fd-4715-9992-a25958dcb538.4341.37:10", 2020-11-19T03:14:40.368 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.0", 2020-11-19T03:14:40.368 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000027", 2020-11-19T03:14:40.368 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status", 2020-11-19T03:14:40.368 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000019", 2020-11-19T03:14:40.369 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:5", 2020-11-19T03:14:40.369 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000039", 2020-11-19T03:14:40.369 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000100", 2020-11-19T03:14:40.369 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-12:d71482d6-15fd-4715-9992-a25958dcb538.4341.12:7", 2020-11-19T03:14:40.369 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000093", 2020-11-19T03:14:40.370 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000004", 2020-11-19T03:14:40.370 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000063", 2020-11-19T03:14:40.370 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000122", 2020-11-19T03:14:40.370 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-16:d71482d6-15fd-4715-9992-a25958dcb538.4341.16:1", 2020-11-19T03:14:40.370 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000057", 2020-11-19T03:14:40.371 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:7", 2020-11-19T03:14:40.371 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000054", 2020-11-19T03:14:40.371 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000114", 2020-11-19T03:14:40.371 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.0", 2020-11-19T03:14:40.371 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000012", 2020-11-19T03:14:40.372 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:1", 2020-11-19T03:14:40.372 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000084", 2020-11-19T03:14:40.372 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000043", 2020-11-19T03:14:40.372 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-14:d71482d6-15fd-4715-9992-a25958dcb538.4341.14:1", 2020-11-19T03:14:40.372 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000111", 2020-11-19T03:14:40.372 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000048", 2020-11-19T03:14:40.373 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000020", 2020-11-19T03:14:40.373 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000099", 2020-11-19T03:14:40.373 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000056", 2020-11-19T03:14:40.373 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-13:d71482d6-15fd-4715-9992-a25958dcb538.4341.13:1", 2020-11-19T03:14:40.374 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000072", 2020-11-19T03:14:40.374 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000062", 2020-11-19T03:14:40.374 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.2.pvIZMfoC8Qr.0", 2020-11-19T03:14:40.374 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.0", 2020-11-19T03:14:40.374 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000115", 2020-11-19T03:14:40.375 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000033", 2020-11-19T03:14:40.375 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000025", 2020-11-19T03:14:40.375 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000123", 2020-11-19T03:14:40.375 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000108", 2020-11-19T03:14:40.375 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000094", 2020-11-19T03:14:40.375 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000060", 2020-11-19T03:14:40.376 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000109", 2020-11-19T03:14:40.376 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000013", 2020-11-19T03:14:40.376 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.1.RdkUwWTRr5c.0", 2020-11-19T03:14:40.376 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-10:d71482d6-15fd-4715-9992-a25958dcb538.4341.10:3", 2020-11-19T03:14:40.376 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:5", 2020-11-19T03:14:40.377 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-19:d71482d6-15fd-4715-9992-a25958dcb538.4341.19:8", 2020-11-19T03:14:40.377 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000051", 2020-11-19T03:14:40.377 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000106", 2020-11-19T03:14:40.377 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000017", 2020-11-19T03:14:40.377 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000023", 2020-11-19T03:14:40.378 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-18:d71482d6-15fd-4715-9992-a25958dcb538.4341.18:1", 2020-11-19T03:14:40.378 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000107", 2020-11-19T03:14:40.378 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000046", 2020-11-19T03:14:40.378 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-23:d71482d6-15fd-4715-9992-a25958dcb538.4341.23:8", 2020-11-19T03:14:40.378 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.full-sync.index.2", 2020-11-19T03:14:40.379 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.2", 2020-11-19T03:14:40.379 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000067", 2020-11-19T03:14:40.379 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000086", 2020-11-19T03:14:40.379 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.0", 2020-11-19T03:14:40.379 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.d71482d6-15fd-4715-9992-a25958dcb538.2", 2020-11-19T03:14:40.380 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-17:d71482d6-15fd-4715-9992-a25958dcb538.4341.17:1", 2020-11-19T03:14:40.380 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000050", 2020-11-19T03:14:40.380 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000102", 2020-11-19T03:14:40.380 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000124", 2020-11-19T03:14:40.380 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-11:d71482d6-15fd-4715-9992-a25958dcb538.4341.11:1", 2020-11-19T03:14:40.381 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000001", 2020-11-19T03:14:40.381 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000126", 2020-11-19T03:14:40.381 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000055", 2020-11-19T03:14:40.381 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000061", 2020-11-19T03:14:40.381 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000097", 2020-11-19T03:14:40.382 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000042", 2020-11-19T03:14:40.382 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.d71482d6-15fd-4715-9992-a25958dcb538:atsuux-26:d71482d6-15fd-4715-9992-a25958dcb538.4341.26:8", 2020-11-19T03:14:40.382 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000075", 2020-11-19T03:14:40.382 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000120", 2020-11-19T03:14:40.382 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000030", 2020-11-19T03:14:40.383 INFO:teuthology.orchestra.run.smithi160.stdout: "data.full-sync.index.d71482d6-15fd-4715-9992-a25958dcb538.3" 2020-11-19T03:14:40.383 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:14:40.383 INFO:teuthology.orchestra.run.smithi160.stdout: 2020-11-19T03:14:40.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:40.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:40.385 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:40.435 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:40.435 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:40.717 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:14:40.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:40.717 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-19T03:14:40.717 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.092660Z" 2020-11-19T03:14:40.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:40.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:40.718 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-19T03:14:40.718 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.095258Z" 2020-11-19T03:14:40.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:40.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:40.719 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-19T03:14:40.719 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.098163Z" 2020-11-19T03:14:40.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:40.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:40.720 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-11-19T03:14:40.720 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.100953Z" 2020-11-19T03:14:40.720 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:40.720 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:14:40.721 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:14:40.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:40.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:40.723 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:40.769 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:40.770 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:41.036 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:41.036 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:41.036 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:41.037 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:41.037 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:41.037 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16121403893923359777 2020-11-19T03:14:41.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:41.037 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:41.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:41.038 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:41.038 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:41.038 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:41.039 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006096", 2020-11-19T03:14:41.039 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:41.039 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:41.039 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:41.039 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:10:08.984297Z" 2020-11-19T03:14:41.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:41.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:41.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:41.040 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:41.043 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:41.044 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:41.044 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:14:41.044 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:41.045 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:41.045 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:41.045 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:14:41.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:41.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:41.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:41.046 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:41.047 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:41.047 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:41.048 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:14:41.048 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:41.048 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:41.049 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:41.049 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:14:41.049 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:41.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:41.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:41.050 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:41.053 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:41.053 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:41.053 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:14:41.053 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:41.054 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:41.054 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:41.054 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:14:41.054 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:41.054 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:41.055 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:41.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:41.055 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:41.055 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:14:41.055 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:14:41.056 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:41.056 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:41.057 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000008472 target=00000000000000000000:00000000000000006096, shard=1 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=2 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=3 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835 2020-11-19T03:14:46.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:46.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:46.232 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:46.278 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:46.278 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:46.546 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:46.546 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:46.547 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:46.547 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:46.547 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:46.547 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16121403893923359777 2020-11-19T03:14:46.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:46.548 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:46.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:46.548 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:46.548 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:46.548 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:46.549 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006096", 2020-11-19T03:14:46.549 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:46.549 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:46.549 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:46.549 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:10:08.984297Z" 2020-11-19T03:14:46.550 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:46.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:46.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:46.550 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:46.550 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:46.551 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:46.551 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-19T03:14:46.551 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:46.551 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:46.551 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:46.552 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:04:00.372839Z" 2020-11-19T03:14:46.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:46.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:46.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:46.552 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:46.553 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:46.553 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:46.553 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-19T03:14:46.553 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:46.554 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:46.554 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:46.555 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:59:14.284797Z" 2020-11-19T03:14:46.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:46.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:46.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:46.556 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:46.556 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:46.556 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:46.556 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-19T03:14:46.556 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:46.557 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:46.557 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:46.557 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:06:35.770852Z" 2020-11-19T03:14:46.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:46.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:46.558 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:46.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:46.558 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:46.558 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:14:46.558 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:14:46.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:46.559 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:46.560 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000008472 target=00000000000000000000:00000000000000006096, shard=1 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=2 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=3 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835 2020-11-19T03:14:51.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:52.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:52.243 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:52.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:52.287 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:52.548 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:52.552 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:52.552 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:52.553 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:52.553 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:52.553 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16121403893923359777 2020-11-19T03:14:52.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:52.554 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:52.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:52.555 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:52.555 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:52.555 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:52.556 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006492", 2020-11-19T03:14:52.556 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:52.558 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:52.558 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:52.559 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:37.143694Z" 2020-11-19T03:14:52.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:52.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:52.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:52.561 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:52.561 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:52.561 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:52.562 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001178", 2020-11-19T03:14:52.562 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:52.564 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:52.565 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:52.565 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:37.147056Z" 2020-11-19T03:14:52.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:52.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:52.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:52.567 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:52.567 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:52.567 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:52.568 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002639", 2020-11-19T03:14:52.568 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:52.570 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:52.571 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:52.571 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:37.134968Z" 2020-11-19T03:14:52.572 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:52.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:52.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:52.573 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:52.573 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:52.574 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:52.574 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000004231", 2020-11-19T03:14:52.576 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:52.576 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:52.576 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:52.577 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:37.140672Z" 2020-11-19T03:14:52.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:52.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:52.578 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:52.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:52.579 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:52.579 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:14:52.579 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:14:52.580 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:52.580 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:52.582 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000008472 target=00000000000000000000:00000000000000006492, shard=1 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000001178, shard=2 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002639, shard=3 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000004231 2020-11-19T03:14:57.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:57.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:57.585 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:57.631 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:57.631 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:57.918 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:57.919 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:57.919 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:57.920 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:57.920 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:57.920 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16121403893923359777 2020-11-19T03:14:57.921 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:57.921 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:57.921 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:57.921 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:57.922 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:57.922 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:57.922 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-19T03:14:57.922 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:57.923 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:57.923 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:57.923 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:38.092660Z" 2020-11-19T03:14:57.923 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:57.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:57.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:57.924 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:57.924 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:57.924 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:57.925 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-19T03:14:57.925 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:57.925 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:57.926 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:57.926 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:38.095258Z" 2020-11-19T03:14:57.926 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:57.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:57.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:57.927 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:57.927 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:57.928 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:57.928 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-19T03:14:57.928 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:57.929 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:57.929 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:57.929 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:38.098163Z" 2020-11-19T03:14:57.929 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:57.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:57.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:57.930 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:57.931 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:57.931 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:57.931 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-11-19T03:14:57.931 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:57.932 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:57.932 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:57.932 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:38.100953Z" 2020-11-19T03:14:57.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:57.933 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:57.933 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:57.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:57.934 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:57.934 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:14:57.934 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:14:57.934 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:57.935 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:57.936 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:14:57.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:57.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:57.938 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:57.978 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:57.978 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:14:58.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:58.248 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-19T03:14:58.248 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.092660Z" 2020-11-19T03:14:58.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:58.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:58.249 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-19T03:14:58.249 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.095258Z" 2020-11-19T03:14:58.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:58.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:58.250 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-19T03:14:58.250 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.098163Z" 2020-11-19T03:14:58.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:58.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:58.250 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-11-19T03:14:58.251 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.100953Z" 2020-11-19T03:14:58.251 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:58.251 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:14:58.252 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:14:58.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:58.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:58.254 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:58.333 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:58.334 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:58.617 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:14:58.617 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:14:58.617 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:14:58.618 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:14:58.618 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:14:58.618 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7000157538207389965 2020-11-19T03:14:58.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:58.619 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:14:58.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:58.619 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:14:58.619 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:14:58.619 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:14:58.620 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-19T03:14:58.620 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:14:58.620 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:14:58.620 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:14:58.620 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.092660Z" 2020-11-19T03:14:58.621 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:58.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:58.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:58.621 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:14:58.622 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:14:58.622 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:14:58.622 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-19T03:14:58.622 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:14:58.622 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:14:58.623 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:14:58.623 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.095258Z" 2020-11-19T03:14:58.623 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:58.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:58.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:58.624 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:14:58.624 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:14:58.624 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:14:58.624 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-19T03:14:58.625 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:14:58.625 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:14:58.625 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:14:58.625 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.098163Z" 2020-11-19T03:14:58.625 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:58.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:58.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:58.626 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:14:58.626 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:14:58.627 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:14:58.627 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-11-19T03:14:58.627 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:14:58.627 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:14:58.627 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:14:58.628 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.100953Z" 2020-11-19T03:14:58.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:58.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:58.628 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:14:58.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:58.629 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:14:58.629 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:14:58.629 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:14:58.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:58.629 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:14:58.631 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:14:58.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:58.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:58.632 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:58.710 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:58.714 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:58.999 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:14:59.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:59.000 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-19T03:14:59.000 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.092660Z" 2020-11-19T03:14:59.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:59.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:59.001 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-19T03:14:59.001 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.095258Z" 2020-11-19T03:14:59.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:59.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:59.002 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-19T03:14:59.002 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.098163Z" 2020-11-19T03:14:59.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:14:59.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:14:59.003 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-11-19T03:14:59.003 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:38.100953Z" 2020-11-19T03:14:59.003 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:14:59.003 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:14:59.004 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:14:59.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:59.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:59.006 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:59.050 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:59.050 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:59.324 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:14:59.325 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:14:59.325 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:14:59.325 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:14:59.325 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:14:59.326 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 12326036592617550149 2020-11-19T03:14:59.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:59.326 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:14:59.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:14:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-19T03:14:59.328 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:59.328 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:59.328 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:59.328 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:38.092660Z" 2020-11-19T03:14:59.329 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:59.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:59.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:59.329 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:14:59.330 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:59.330 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:59.330 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-19T03:14:59.330 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:59.331 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:59.331 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:59.331 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:38.095258Z" 2020-11-19T03:14:59.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:59.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:59.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:59.332 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:14:59.332 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:59.333 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:59.333 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-19T03:14:59.333 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:59.333 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:59.334 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:59.334 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:38.098163Z" 2020-11-19T03:14:59.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:59.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:59.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:59.335 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:14:59.336 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:14:59.336 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:14:59.336 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-11-19T03:14:59.336 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:14:59.337 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:14:59.337 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:14:59.337 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:38.100953Z" 2020-11-19T03:14:59.337 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:59.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:59.338 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:14:59.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:59.338 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:14:59.339 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:14:59.339 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:14:59.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:59.340 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:14:59.341 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:14:59.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:59.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:59.343 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:59.379 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:59.380 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:14:59.686 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:14:59.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:59.688 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006885", 2020-11-19T03:14:59.688 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.496151Z" 2020-11-19T03:14:59.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:59.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:59.690 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-19T03:14:59.691 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.498406Z" 2020-11-19T03:14:59.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:59.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:59.692 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-19T03:14:59.693 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.487983Z" 2020-11-19T03:14:59.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:14:59.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:14:59.694 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-19T03:14:59.695 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.491780Z" 2020-11-19T03:14:59.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:14:59.696 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:14:59.701 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:14:59.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:59.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:14:59.703 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:14:59.754 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:14:59.755 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:00.024 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:15:00.025 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:15:00.025 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:15:00.025 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:15:00.026 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:15:00.026 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7306459910885960034 2020-11-19T03:15:00.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.027 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:15:00.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:00.027 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:15:00.027 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:00.028 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:00.028 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006885", 2020-11-19T03:15:00.028 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:00.028 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:00.029 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:00.029 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.496151Z" 2020-11-19T03:15:00.029 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:00.030 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:15:00.030 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:00.031 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:00.031 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-19T03:15:00.031 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:00.032 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:00.032 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:00.032 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.498406Z" 2020-11-19T03:15:00.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:00.033 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:15:00.033 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:00.034 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:00.034 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-19T03:15:00.038 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:00.038 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:00.038 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:00.038 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.487983Z" 2020-11-19T03:15:00.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:00.039 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:15:00.039 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:00.040 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:00.040 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-19T03:15:00.040 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:00.040 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:00.040 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:00.041 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.491780Z" 2020-11-19T03:15:00.041 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.041 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.041 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:15:00.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.042 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:15:00.042 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:15:00.042 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:15:00.042 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.043 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:15:00.044 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:15:00.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:00.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:00.045 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:00.090 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:00.091 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:00.352 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:15:00.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:00.352 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006885", 2020-11-19T03:15:00.353 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.496151Z" 2020-11-19T03:15:00.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:00.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:00.353 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-19T03:15:00.354 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.498406Z" 2020-11-19T03:15:00.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:00.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:00.354 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-19T03:15:00.354 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.487983Z" 2020-11-19T03:15:00.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:00.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:00.355 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-19T03:15:00.355 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.491780Z" 2020-11-19T03:15:00.356 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:00.356 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:15:00.357 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:15:00.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:00.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:00.358 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:00.656 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:15:00.656 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:15:00.657 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:15:00.657 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:15:00.657 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:15:00.657 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 597164135494128594 2020-11-19T03:15:00.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.658 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:15:00.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:00.658 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:15:00.658 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:00.659 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:00.659 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006885", 2020-11-19T03:15:00.659 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:00.659 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:00.660 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:00.660 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.496151Z" 2020-11-19T03:15:00.660 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:00.661 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:15:00.661 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:00.661 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:00.661 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-19T03:15:00.661 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:00.662 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:00.662 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:00.662 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.498406Z" 2020-11-19T03:15:00.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:00.663 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:15:00.663 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:00.664 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:00.664 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-19T03:15:00.664 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:00.664 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.487983Z" 2020-11-19T03:15:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:15:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-19T03:15:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:00.667 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:00.667 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:00.667 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.491780Z" 2020-11-19T03:15:00.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:15:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:15:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:15:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:15:00.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:00.669 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:15:00.670 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:15:00.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:00.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:00.672 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:00.753 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:15:01.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.053 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006753", 2020-11-19T03:15:01.054 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:41.927225Z" 2020-11-19T03:15:01.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.060 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-19T03:15:01.060 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:41.929837Z" 2020-11-19T03:15:01.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.061 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-19T03:15:01.062 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:41.922315Z" 2020-11-19T03:15:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-11-19T03:15:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:41.924770Z" 2020-11-19T03:15:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:01.066 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:15:01.066 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:15:01.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:01.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:01.068 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:01.136 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:01.136 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:15:01.414 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:15:01.414 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:15:01.414 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:15:01.415 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:15:01.415 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 2448524765768921960 2020-11-19T03:15:01.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.415 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:15:01.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.416 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:15:01.416 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:01.416 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:01.416 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006753", 2020-11-19T03:15:01.416 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:01.417 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:01.417 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:01.417 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:41.927225Z" 2020-11-19T03:15:01.417 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:01.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.418 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:15:01.418 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:01.418 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:01.419 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-19T03:15:01.419 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:01.419 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:01.419 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:01.420 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:41.929837Z" 2020-11-19T03:15:01.420 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:01.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.421 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:15:01.421 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:01.421 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:01.421 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-19T03:15:01.421 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:01.422 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:01.422 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:01.422 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:41.922315Z" 2020-11-19T03:15:01.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:01.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.423 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:15:01.423 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:01.424 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:15:01.424 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-11-19T03:15:01.424 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:01.424 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:01.424 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:01.425 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:41.924770Z" 2020-11-19T03:15:01.425 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:01.425 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:01.425 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:15:01.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.426 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:15:01.426 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:15:01.426 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:15:01.426 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:01.427 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:15:01.428 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:15:01.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:01.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:01.429 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:01.507 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:01.507 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:01.787 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:15:01.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.788 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006753", 2020-11-19T03:15:01.788 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:41.927225Z" 2020-11-19T03:15:01.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.789 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-19T03:15:01.789 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:41.929837Z" 2020-11-19T03:15:01.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.790 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-19T03:15:01.790 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:41.922315Z" 2020-11-19T03:15:01.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:01.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:01.791 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-11-19T03:15:01.791 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:14:41.924770Z" 2020-11-19T03:15:01.791 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:01.792 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:15:01.793 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:15:01.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:01.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:01.794 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:01.845 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:01.845 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:02.127 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:15:02.842 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:15:02.842 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:15:02.842 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:15:02.843 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:15:02.843 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16483361435957904979 2020-11-19T03:15:02.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:02.843 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:15:02.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:02.844 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:15:02.844 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:02.844 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:15:02.844 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006753", 2020-11-19T03:15:02.844 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:02.845 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:02.845 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:02.845 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:41.927225Z" 2020-11-19T03:15:02.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:02.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:02.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:02.849 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:15:02.850 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:02.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:15:02.856 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-19T03:15:02.857 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:02.858 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:02.858 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:02.860 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:41.929837Z" 2020-11-19T03:15:02.861 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:02.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:02.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:02.862 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:15:02.862 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:02.862 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:15:02.863 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-19T03:15:02.863 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:02.864 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:02.864 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:02.864 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:41.922315Z" 2020-11-19T03:15:02.865 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:02.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:02.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:02.866 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:15:02.866 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:02.866 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:15:02.867 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-11-19T03:15:02.867 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:02.867 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:02.870 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:02.870 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:14:41.924770Z" 2020-11-19T03:15:02.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:02.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:02.871 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:15:02.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:02.872 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:15:02.872 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:15:02.873 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:15:02.873 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:02.873 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:15:02.876 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:15:02.878 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2020-11-19T03:15:02.880 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-59 2020-11-19T03:15:02.932 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-60 2020-11-19T03:15:03.003 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-61 2020-11-19T03:15:03.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:03.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:03.723 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:03.765 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:03.765 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:03.998 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:15:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:15:04.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:04.037 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755703.063645_764.1", 2020-11-19T03:15:04.037 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:03.063645Z" 2020-11-19T03:15:04.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:04.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:04.038 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:15:04.038 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:15:04.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:04.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:04.039 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:15:04.039 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:15:04.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:04.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:04.040 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:15:04.040 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:15:04.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:04.041 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:15:04.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:15:04.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:04.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:04.044 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:04.087 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:04.088 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:04.331 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:15:04.331 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:15:04.332 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:15:04.332 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:15:04.332 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:15:04.332 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:15:04.333 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:15:04.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.333 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:15:04.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:04.334 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:15:04.334 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:04.334 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:15:04.334 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755703.063645_764.1", 2020-11-19T03:15:04.334 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:04.335 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:15:04.335 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:04.335 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:03.063645Z", 2020-11-19T03:15:04.335 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:15:04.336 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:04.336 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:15:04.336 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:04.337 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:15:04.337 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:15:04.337 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:04.337 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:04.338 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:04.338 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:15:04.338 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:15:04.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:04.339 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:15:04.339 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:04.339 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:15:04.340 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:15:04.340 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:04.340 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:04.340 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:04.341 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:15:04.341 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:15:04.341 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:04.342 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:15:04.342 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:04.342 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:15:04.342 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:15:04.343 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:04.343 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:04.343 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:04.343 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:15:04.344 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:15:04.344 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.344 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.344 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:15:04.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.345 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:15:04.345 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:15:04.345 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:15:04.345 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.346 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:15:04.347 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:15:04.347 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:15:04.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:04.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:04.349 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:15:04.635 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:15:04.635 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:15:04.636 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:15:04.636 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:15:04.636 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:15:04.636 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:15:04.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:04.637 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:15:04.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:04.637 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:15:04.637 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:04.638 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:15:04.638 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755703.063645_764.1", 2020-11-19T03:15:04.638 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:04.638 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:15:04.639 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:04.639 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:03.063645Z", 2020-11-19T03:15:04.639 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:15:04.639 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:04.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:04.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:04.640 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:15:04.640 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:04.641 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:15:04.641 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:15:04.641 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:04.641 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:04.642 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:04.642 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:15:04.642 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:15:04.642 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:15:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:15:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:15:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:15:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:15:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:15:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:15:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:15:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:15:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:15:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:15:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:15:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:15:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:15:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:15:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:15:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:15:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:15:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:15:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:15:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:15:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:15:04.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:15:04.651 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:15:04.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:04.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:15:04.652 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:15:04.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:04.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:04.962 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:15:04.963 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:15:04.963 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:15:04.963 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:15:04.963 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:15:04.964 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:15:04.964 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:15:04.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.964 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:15:04.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:04.965 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:15:04.965 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:04.965 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:15:04.965 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755703.063645_764.1", 2020-11-19T03:15:04.965 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:04.966 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:15:04.966 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:04.966 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:03.063645Z", 2020-11-19T03:15:04.966 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:15:04.966 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:04.967 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:15:04.967 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:04.968 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:15:04.968 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:15:04.968 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:04.968 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:04.969 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:04.969 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:15:04.969 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:15:04.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:04.970 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:15:04.970 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:04.970 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:15:04.970 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:15:04.971 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:04.971 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:04.971 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:04.971 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:15:04.971 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:15:04.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:15:04.972 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:15:04.972 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:15:04.973 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:15:04.973 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:15:04.973 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:15:04.973 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:15:04.973 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:15:04.974 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:15:04.974 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:15:04.974 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.975 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:15:04.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:15:04.975 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:15:04.975 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:15:04.976 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:15:04.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:15:04.976 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:15:04.976 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:15:04.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:04.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:04.978 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:05.033 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:05.033 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:05.263 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:05.257+0000 7effdb9da0c0 1 bucket sync caught up with source: 2020-11-19T03:15:05.263 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.33.5, , 00000000001.36.5, , 00000000001.71.5, , , 00000000001.36.5, , , ] 2020-11-19T03:15:05.263 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.33.5, , 00000000001.36.5, , 00000000001.71.5, , , 00000000001.36.5, , , ] 2020-11-19T03:15:05.263 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:05.257+0000 7effdb9da0c0 0 bucket checkpoint complete 2020-11-19T03:15:05.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:05.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:05.299 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:05.605 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:05.600+0000 7f2dac8cd0c0 1 bucket sync caught up with source: 2020-11-19T03:15:05.605 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.33.5, , 00000000001.36.5, , 00000000001.71.5, , , 00000000001.36.5, , , ] 2020-11-19T03:15:05.606 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.33.5, , 00000000001.36.5, , 00000000001.71.5, , , 00000000001.36.5, , , ] 2020-11-19T03:15:05.606 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:05.600+0000 7f2dac8cd0c0 0 bucket checkpoint complete 2020-11-19T03:15:05.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:05.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:05.633 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:05.681 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:05.682 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:05.928 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:05.921+0000 7fba94da30c0 1 bucket sync caught up with source: 2020-11-19T03:15:05.929 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.33.5, , 00000000001.36.5, , 00000000001.71.5, , , 00000000001.36.5, , , ] 2020-11-19T03:15:05.929 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.33.5, , 00000000001.36.5, , 00000000001.71.5, , , 00000000001.36.5, , , ] 2020-11-19T03:15:05.929 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:05.921+0000 7fba94da30c0 0 bucket checkpoint complete 2020-11-19T03:15:06.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:06.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:06.051 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:06.103 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:06.104 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:06.361 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:06.356+0000 7f445ebca0c0 1 bucket sync caught up with source: 2020-11-19T03:15:06.362 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.32.5, , 00000000001.46.5, , 00000000001.66.5, , , 00000000001.56.5, , , ] 2020-11-19T03:15:06.362 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.32.5, , 00000000001.46.5, , 00000000001.66.5, , , 00000000001.56.5, , , ] 2020-11-19T03:15:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:06.356+0000 7f445ebca0c0 0 bucket checkpoint complete 2020-11-19T03:15:06.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:06.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:06.397 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:06.488 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:06.488 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:06.732 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:06.724+0000 7f03f436a0c0 1 bucket sync caught up with source: 2020-11-19T03:15:06.733 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.32.5, , 00000000001.46.5, , 00000000001.66.5, , , 00000000001.56.5, , , ] 2020-11-19T03:15:06.733 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.32.5, , 00000000001.46.5, , 00000000001.66.5, , , 00000000001.56.5, , , ] 2020-11-19T03:15:06.733 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:06.724+0000 7f03f436a0c0 0 bucket checkpoint complete 2020-11-19T03:15:06.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:06.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:06.768 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:07.073 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:07.064+0000 7f1dee4e00c0 1 bucket sync caught up with source: 2020-11-19T03:15:07.073 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.29.5, , 00000000001.40.5, , 00000000001.61.5, , , 00000000001.33.5, , , ] 2020-11-19T03:15:07.074 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.29.5, , 00000000001.40.5, , 00000000001.61.5, , , 00000000001.33.5, , , ] 2020-11-19T03:15:07.074 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:07.064+0000 7f1dee4e00c0 0 bucket checkpoint complete 2020-11-19T03:15:07.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:07.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:07.097 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:07.160 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:07.161 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:07.424 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:07.417+0000 7f871d4f40c0 1 bucket sync caught up with source: 2020-11-19T03:15:08.174 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.40.5, , 00000000001.61.5, , , 00000000001.33.5, , , ] 2020-11-19T03:15:08.174 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.40.5, , 00000000001.61.5, , , 00000000001.33.5, , , ] 2020-11-19T03:15:08.174 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:07.417+0000 7f871d4f40c0 0 bucket checkpoint complete 2020-11-19T03:15:08.175 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-59 zones={test-zone2, test-zone1} 2020-11-19T03:15:08.188 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:15:08.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:08.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:08.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:08.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:08.202 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:15:08.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:08.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:08.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:08.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:08.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:08.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:08.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:08.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:08.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:08.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:08.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:08.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:09.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:09.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:09.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:09.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:09.410 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-59 zones={test-zone2, test-zone1} 2020-11-19T03:15:09.411 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-59 zones={test-zone3, test-zone1} 2020-11-19T03:15:09.421 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:15:09.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:09.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:09.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:09.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:09.430 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:15:09.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:09.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:09.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:09.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:09.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:09.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:09.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:09.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:09.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:09.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:10.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:10.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:10.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:10.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:10.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:10.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:10.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-59 zones={test-zone3, test-zone1} 2020-11-19T03:15:10.642 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-59 zones={test-zone3, test-zone2} 2020-11-19T03:15:10.658 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:15:10.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:10.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:10.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:10.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:10.666 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:15:10.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:10.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:10.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:10.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:10.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:10.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:10.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:10.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:11.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:11.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:11.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:11.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:11.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:11.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:11.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:11.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:11.934 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-59 zones={test-zone3, test-zone2} 2020-11-19T03:15:11.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:11.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:11.936 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:11.991 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:11.991 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:12.228 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:12.221+0000 7fb4f7b5e0c0 1 bucket sync caught up with source: 2020-11-19T03:15:12.228 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.38.5, , 00000000001.33.5, , 00000000001.30.5, , , 00000000001.57.5, , , ] 2020-11-19T03:15:12.228 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.38.5, , 00000000001.33.5, , 00000000001.30.5, , , 00000000001.57.5, , , ] 2020-11-19T03:15:12.229 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:12.221+0000 7fb4f7b5e0c0 0 bucket checkpoint complete 2020-11-19T03:15:12.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:12.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:12.260 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:12.310 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:12.311 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:12.556+0000 7fba1e0600c0 1 bucket sync caught up with source: 2020-11-19T03:15:12.565 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.38.5, , 00000000001.33.5, , 00000000001.30.5, , , 00000000001.57.5, , , ] 2020-11-19T03:15:12.565 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.38.5, , 00000000001.33.5, , 00000000001.30.5, , , 00000000001.57.5, , , ] 2020-11-19T03:15:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:12.556+0000 7fba1e0600c0 0 bucket checkpoint complete 2020-11-19T03:15:12.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:12.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:12.594 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:12.641 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:12.642 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:12.902 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:12.893+0000 7f2542ebf0c0 1 bucket sync caught up with source: 2020-11-19T03:15:12.902 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.38.5, , 00000000001.33.5, , 00000000001.30.5, , , 00000000001.57.5, , , ] 2020-11-19T03:15:12.902 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.38.5, , 00000000001.33.5, , 00000000001.30.5, , , 00000000001.57.5, , , ] 2020-11-19T03:15:12.902 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:12.893+0000 7f2542ebf0c0 0 bucket checkpoint complete 2020-11-19T03:15:13.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:13.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:13.025 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:13.077 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:13.078 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:13.348+0000 7f7b8add60c0 1 bucket sync caught up with source: 2020-11-19T03:15:13.957 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.44.5, , 00000000001.43.5, , 00000000001.34.5, , , 00000000001.65.5, , , ] 2020-11-19T03:15:13.958 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.44.5, , 00000000001.43.5, , 00000000001.34.5, , , 00000000001.65.5, , , ] 2020-11-19T03:15:13.958 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:13.348+0000 7f7b8add60c0 0 bucket checkpoint complete 2020-11-19T03:15:13.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:13.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:13.963 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:14.052 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:14.052 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:14.293 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:14.288+0000 7f8880b220c0 1 bucket sync caught up with source: 2020-11-19T03:15:14.294 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.44.5, , 00000000001.43.5, , 00000000001.34.5, , , 00000000001.65.5, , , ] 2020-11-19T03:15:14.294 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.44.5, , 00000000001.43.5, , 00000000001.34.5, , , 00000000001.65.5, , , ] 2020-11-19T03:15:14.294 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:14.288+0000 7f8880b220c0 0 bucket checkpoint complete 2020-11-19T03:15:14.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:14.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:14.329 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:14.421 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:14.422 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:14.687 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:14.680+0000 7f2a76b930c0 1 bucket sync caught up with source: 2020-11-19T03:15:14.687 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.42.5, , 00000000001.36.5, , 00000000001.32.5, , , 00000000001.85.5, , , ] 2020-11-19T03:15:14.687 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.42.5, , 00000000001.36.5, , 00000000001.32.5, , , 00000000001.85.5, , , ] 2020-11-19T03:15:14.688 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:14.680+0000 7f2a76b930c0 0 bucket checkpoint complete 2020-11-19T03:15:14.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:14.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:14.716 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:14.766 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:14.766 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:15.035 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:15.029+0000 7fbe1c4f90c0 1 bucket sync caught up with source: 2020-11-19T03:15:15.035 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.42.5, , 00000000001.36.5, , 00000000001.32.5, , , 00000000001.85.5, , , ] 2020-11-19T03:15:15.036 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.42.5, , 00000000001.36.5, , 00000000001.32.5, , , 00000000001.85.5, , , ] 2020-11-19T03:15:15.036 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:15.029+0000 7fbe1c4f90c0 0 bucket checkpoint complete 2020-11-19T03:15:15.067 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-60 zones={test-zone2, test-zone1} 2020-11-19T03:15:15.083 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:15:15.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:15.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:15.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:15.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:15.093 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:15:15.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:15.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:15.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:15.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:15.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:15.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:15.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:15.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:15.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:15.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:15.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:15.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:15.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:15.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:16.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:16.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:16.306 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-60 zones={test-zone2, test-zone1} 2020-11-19T03:15:16.306 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-60 zones={test-zone3, test-zone1} 2020-11-19T03:15:16.322 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:15:16.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:16.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:16.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:16.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:16.336 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:15:16.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:16.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:16.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:16.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:16.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:16.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:16.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:16.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:16.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:16.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:16.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:16.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:17.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:17.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:17.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:17.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:17.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-60 zones={test-zone3, test-zone1} 2020-11-19T03:15:17.538 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-60 zones={test-zone3, test-zone2} 2020-11-19T03:15:17.550 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:15:17.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:17.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:17.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:17.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:17.558 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:15:17.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:17.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:17.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:17.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:17.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:17.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:17.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:17.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:17.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:18.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:18.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:18.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:18.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:18.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:18.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:18.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:18.757 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-60 zones={test-zone3, test-zone2} 2020-11-19T03:15:18.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:18.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:18.759 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:18.803 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:18.804 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:19.060 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:19.053+0000 7f48426e00c0 1 bucket sync caught up with source: 2020-11-19T03:15:19.060 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.85.5, , 00000000001.71.5, , , 00000000001.26.5, , , ] 2020-11-19T03:15:19.060 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.85.5, , 00000000001.71.5, , , 00000000001.26.5, , , ] 2020-11-19T03:15:19.061 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:19.053+0000 7f48426e00c0 0 bucket checkpoint complete 2020-11-19T03:15:19.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:19.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:19.095 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:19.142 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:19.142 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:19.388+0000 7f7b7ce760c0 1 bucket sync caught up with source: 2020-11-19T03:15:19.400 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.29.5, , 00000000001.85.5, , 00000000001.71.5, , , 00000000001.26.5, , , ] 2020-11-19T03:15:19.400 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.29.5, , 00000000001.85.5, , 00000000001.71.5, , , 00000000001.26.5, , , ] 2020-11-19T03:15:19.400 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:19.388+0000 7f7b7ce760c0 0 bucket checkpoint complete 2020-11-19T03:15:19.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:19.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:19.425 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:19.479 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:19.480 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:19.720 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:19.713+0000 7f072066f0c0 1 bucket sync caught up with source: 2020-11-19T03:15:19.721 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.85.5, , 00000000001.71.5, , , 00000000001.26.5, , , ] 2020-11-19T03:15:19.721 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.85.5, , 00000000001.71.5, , , 00000000001.26.5, , , ] 2020-11-19T03:15:19.721 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:19.713+0000 7f072066f0c0 0 bucket checkpoint complete 2020-11-19T03:15:19.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:19.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:19.846 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:19.897 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:19.897 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:20.144+0000 7f7d26edc0c0 1 bucket sync caught up with source: 2020-11-19T03:15:20.150 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.31.5, , 00000000001.89.5, , 00000000001.75.5, , , 00000000001.24.5, , , ] 2020-11-19T03:15:20.151 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.31.5, , 00000000001.89.5, , 00000000001.75.5, , , 00000000001.24.5, , , ] 2020-11-19T03:15:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:20.144+0000 7f7d26edc0c0 0 bucket checkpoint complete 2020-11-19T03:15:20.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:20.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:20.183 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:20.231 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:20.231 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:20.495 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:20.488+0000 7f72d03030c0 1 bucket sync caught up with source: 2020-11-19T03:15:20.496 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.31.5, , 00000000001.89.5, , 00000000001.75.5, , , 00000000001.24.5, , , ] 2020-11-19T03:15:20.496 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.31.5, , 00000000001.89.5, , 00000000001.75.5, , , 00000000001.24.5, , , ] 2020-11-19T03:15:20.496 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:20.488+0000 7f72d03030c0 0 bucket checkpoint complete 2020-11-19T03:15:20.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:20.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:20.526 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:20.613 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:20.613 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:20.868 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:20.860+0000 7fd46bb0d0c0 1 bucket sync caught up with source: 2020-11-19T03:15:20.868 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.33.5, , 00000000001.101.5, , 00000000001.73.5, , , 00000000001.23.5, , , ] 2020-11-19T03:15:20.868 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.33.5, , 00000000001.101.5, , 00000000001.73.5, , , 00000000001.23.5, , , ] 2020-11-19T03:15:20.869 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:20.860+0000 7fd46bb0d0c0 0 bucket checkpoint complete 2020-11-19T03:15:20.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:20.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:20.896 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:20.946 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:20.946 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:21.209 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:21.201+0000 7fdb78a8d0c0 1 bucket sync caught up with source: 2020-11-19T03:15:21.209 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.33.5, , 00000000001.101.5, , 00000000001.73.5, , , 00000000001.23.5, , , ] 2020-11-19T03:15:21.209 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.33.5, , 00000000001.101.5, , 00000000001.73.5, , , 00000000001.23.5, , , ] 2020-11-19T03:15:21.210 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:21.201+0000 7fdb78a8d0c0 0 bucket checkpoint complete 2020-11-19T03:15:21.239 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-61 zones={test-zone2, test-zone1} 2020-11-19T03:15:21.249 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:15:21.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:21.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:21.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:21.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:21.256 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:15:21.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:21.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:21.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:21.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:21.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:21.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:21.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:21.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:21.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:21.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:21.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:21.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:22.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:22.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:22.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:22.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:22.500 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-61 zones={test-zone2, test-zone1} 2020-11-19T03:15:22.502 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-61 zones={test-zone3, test-zone1} 2020-11-19T03:15:22.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:15:22.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:22.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:22.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:22.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:22.546 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:15:22.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:22.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:22.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:22.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:22.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:22.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:22.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:22.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:23.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:23.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:23.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:23.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:23.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:23.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:23.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:23.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:23.858 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-61 zones={test-zone3, test-zone1} 2020-11-19T03:15:23.859 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-61 zones={test-zone3, test-zone2} 2020-11-19T03:15:23.871 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:15:23.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:23.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:23.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:23.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:23.879 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:15:23.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:15:23.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:15:23.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:15:23.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:15:23.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:23.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:24.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:15:24.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:24.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:24.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:15:24.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:25.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:25.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:15:25.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:25.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:25.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:15:26.107 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-61 zones={test-zone3, test-zone2} 2020-11-19T03:15:26.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'atsuux-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:26.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'atsuux-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:26.109 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket atsuux-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:15:26.154 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:26.154 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:26.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'atsuux-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:26.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'atsuux-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:26.498 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket atsuux-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:15:26.579 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:26.580 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:26.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'atsuux-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:26.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'atsuux-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:26.987 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket atsuux-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:15:27.073 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:27.076 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:27.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'atsuux-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:27.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'atsuux-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:27.406 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket atsuux-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:15:27.451 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:27.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'atsuux-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:27.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'atsuux-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:27.790 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket atsuux-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:15:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:27.867 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:28.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'atsuux-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:28.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'atsuux-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:15:28.199 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket atsuux-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:15:28.289 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:28.289 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:29.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:29.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:29.312 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:29.358 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:29.358 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:29.621 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:29.613+0000 7f4b09b730c0 1 waiting for incremental sync to catch up: 2020-11-19T03:15:29.621 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.37.1, , 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, , 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, , 00000000004.79.1] 2020-11-19T03:15:29.621 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:34.626 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:34.621+0000 7f4b09b730c0 1 waiting for incremental sync to catch up: 2020-11-19T03:15:34.627 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.37.1, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:34.628 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:39.631 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:39.625+0000 7f4b09b730c0 1 waiting for incremental sync to catch up: 2020-11-19T03:15:39.631 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.37.1, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:39.632 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:44.636 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:44.629+0000 7f4b09b730c0 1 waiting for incremental sync to catch up: 2020-11-19T03:15:44.636 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000005.37.1, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:44.636 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:49.642 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:49.633+0000 7f4b09b730c0 1 bucket sync caught up with source: 2020-11-19T03:15:49.642 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:49.643 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:49.643 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:49.633+0000 7f4b09b730c0 0 bucket checkpoint complete 2020-11-19T03:15:49.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:49.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:49.669 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:49.726 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:49.726 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:49.984 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:49.979+0000 7fe1d2ada0c0 1 bucket sync caught up with source: 2020-11-19T03:15:49.985 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:49.985 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:49.985 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:49.979+0000 7fe1d2ada0c0 0 bucket checkpoint complete 2020-11-19T03:15:50.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:50.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:50.015 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:50.062 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:50.063 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:50.300 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:50.293+0000 7f48735e40c0 1 bucket sync caught up with source: 2020-11-19T03:15:50.301 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:50.301 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.39.5, 00000000004.51.1, 00000000006.48.5, 00000000004.54.1, 00000000006.81.5, 00000000004.35.1, 00000000004.46.1, 00000000006.42.5, 00000000004.33.1, 00000000004.48.1, 00000000004.79.1] 2020-11-19T03:15:50.301 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:15:50.293+0000 7f48735e40c0 0 bucket checkpoint complete 2020-11-19T03:15:50.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:50.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:15:50.432 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:15:50.482 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:15:50.483 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:15:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:50.731+0000 7efd396b40c0 1 waiting for incremental sync to catch up: 2020-11-19T03:15:50.736 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.32.5, , 00000000001.46.5, , 00000000001.66.5, , , 00000000002.58.5, , , ] 2020-11-19T03:15:50.736 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.34.5, , 00000000002.49.5, , 00000000002.68.5, , , 00000000002.58.5, , , ] 2020-11-19T03:15:55.740 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:15:55.735+0000 7efd396b40c0 1 waiting for incremental sync to catch up: 2020-11-19T03:15:55.921 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.32.5, , 00000000001.46.5, , 00000000001.66.5, , , 00000000002.58.5, , , ] 2020-11-19T03:15:55.921 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.34.5, , 00000000002.49.5, , 00000000002.68.5, , , 00000000002.58.5, , , ] 2020-11-19T03:16:00.745 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:00.739+0000 7efd396b40c0 1 bucket sync caught up with source: 2020-11-19T03:16:00.745 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000002.34.5, , 00000000002.49.5, , 00000000002.68.5, , , 00000000002.58.5, , , ] 2020-11-19T03:16:00.746 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.34.5, , 00000000002.49.5, , 00000000002.68.5, , , 00000000002.58.5, , , ] 2020-11-19T03:16:00.746 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:00.739+0000 7efd396b40c0 0 bucket checkpoint complete 2020-11-19T03:16:00.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:00.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:00.781 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:00.829 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:00.830 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:01.091+0000 7fdfa7e400c0 1 waiting for incremental sync to catch up: 2020-11-19T03:16:01.097 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.32.5, , 00000000001.46.5, , 00000000001.66.5, , , 00000000002.58.5, , , ] 2020-11-19T03:16:01.097 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.34.5, , 00000000002.49.5, , 00000000002.68.5, , , 00000000002.58.5, , , ] 2020-11-19T03:16:06.100 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:06.095+0000 7fdfa7e400c0 1 bucket sync caught up with source: 2020-11-19T03:16:06.101 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000002.34.5, , 00000000002.49.5, , 00000000002.68.5, , , 00000000002.58.5, , , ] 2020-11-19T03:16:06.101 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.34.5, , 00000000002.49.5, , 00000000002.68.5, , , 00000000002.58.5, , , ] 2020-11-19T03:16:06.101 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:06.095+0000 7fdfa7e400c0 0 bucket checkpoint complete 2020-11-19T03:16:06.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:06.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:06.131 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:06.222 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:06.222 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:06.472 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:06.467+0000 7facb37960c0 1 bucket sync caught up with source: 2020-11-19T03:16:06.472 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000002.33.5, , 00000000002.48.5, , 00000000002.67.5, , , 00000000002.37.5, , , ] 2020-11-19T03:16:06.473 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.33.5, , 00000000002.48.5, , 00000000002.67.5, , , 00000000002.37.5, , , ] 2020-11-19T03:16:06.473 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:06.467+0000 7facb37960c0 0 bucket checkpoint complete 2020-11-19T03:16:06.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:06.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:06.502 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:06.556 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:06.557 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:06.810 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:06.801+0000 7f32838700c0 1 bucket sync caught up with source: 2020-11-19T03:16:06.810 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.33.5, , 00000000002.48.5, , 00000000002.67.5, , , 00000000002.37.5, , , ] 2020-11-19T03:16:06.811 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.33.5, , 00000000002.48.5, , 00000000002.67.5, , , 00000000002.37.5, , , ] 2020-11-19T03:16:06.811 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:06.801+0000 7f32838700c0 0 bucket checkpoint complete 2020-11-19T03:16:06.844 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-59 zones={test-zone2, test-zone1} 2020-11-19T03:16:06.864 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:06.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:06.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:06.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:06.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:06.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:06.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:06.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:06.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:06.878 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:06.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:06.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:06.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:06.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:06.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:06.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:06.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:06.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:06.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:07.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:07.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:07.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:07.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:07.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:07.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:07.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:07.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:07.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:07.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:08.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:08.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:08.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:08.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:08.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:08.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:08.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:08.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:08.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:09.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:09.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:09.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:09.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:09.492 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-59 zones={test-zone2, test-zone1} 2020-11-19T03:16:09.495 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-59 zones={test-zone3, test-zone1} 2020-11-19T03:16:09.512 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:09.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:09.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:09.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:09.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:09.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:09.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:09.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:09.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:09.526 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:09.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:09.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:09.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:09.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:09.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:09.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:09.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:09.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:09.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:09.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:09.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:09.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:09.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:10.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:10.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:10.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:10.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:10.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:10.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:10.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:10.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:10.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:10.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:11.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:11.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:11.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:11.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:11.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:11.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:12.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:12.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:12.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:12.318 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-59 zones={test-zone3, test-zone1} 2020-11-19T03:16:12.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-59 zones={test-zone3, test-zone2} 2020-11-19T03:16:12.329 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:12.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:12.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:12.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:12.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:12.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:12.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:12.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:12.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:12.341 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:12.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:12.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:12.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:12.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:12.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:12.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:12.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:12.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:12.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:12.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:12.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:12.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:12.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:12.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:13.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:13.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:13.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:13.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:13.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:13.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:13.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:13.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:13.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:13.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:14.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:14.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:14.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:14.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:14.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:14.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:14.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:14.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:15.020 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-59 zones={test-zone3, test-zone2} 2020-11-19T03:16:15.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:15.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:15.024 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:15.078 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:15.078 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:15.325 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:15.317+0000 7f3d209320c0 1 bucket sync caught up with source: 2020-11-19T03:16:15.325 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.50.5, 00000000004.40.1, 00000000006.43.5, 00000000004.61.1, 00000000006.44.5, 00000000004.59.1, 00000000004.64.1, 00000000006.71.5, 00000000004.50.1, 00000000004.37.1, 00000000004.38.1] 2020-11-19T03:16:15.326 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.50.5, 00000000004.40.1, 00000000006.43.5, 00000000004.61.1, 00000000006.44.5, 00000000004.59.1, 00000000004.64.1, 00000000006.71.5, 00000000004.50.1, 00000000004.37.1, 00000000004.38.1] 2020-11-19T03:16:15.326 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:15.317+0000 7f3d209320c0 0 bucket checkpoint complete 2020-11-19T03:16:15.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:15.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:15.359 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:15.406 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:15.406 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:15.675 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:15.671+0000 7f571568d0c0 1 bucket sync caught up with source: 2020-11-19T03:16:15.675 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000006.50.5, 00000000004.40.1, 00000000006.43.5, 00000000004.61.1, 00000000006.44.5, 00000000004.59.1, 00000000004.64.1, 00000000006.71.5, 00000000004.50.1, 00000000004.37.1, 00000000004.38.1] 2020-11-19T03:16:15.675 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000006.50.5, 00000000004.40.1, 00000000006.43.5, 00000000004.61.1, 00000000006.44.5, 00000000004.59.1, 00000000004.64.1, 00000000006.71.5, 00000000004.50.1, 00000000004.37.1, 00000000004.38.1] 2020-11-19T03:16:15.676 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:15.671+0000 7f571568d0c0 0 bucket checkpoint complete 2020-11-19T03:16:15.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:15.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:15.707 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:15.761 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:15.762 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:16.007 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:16.001+0000 7f256be6a0c0 1 bucket sync caught up with source: 2020-11-19T03:16:16.008 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.50.5, 00000000004.40.1, 00000000006.43.5, 00000000004.61.1, 00000000006.44.5, 00000000004.59.1, 00000000004.64.1, 00000000006.71.5, 00000000004.50.1, 00000000004.37.1, 00000000004.38.1] 2020-11-19T03:16:16.008 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.50.5, 00000000004.40.1, 00000000006.43.5, 00000000004.61.1, 00000000006.44.5, 00000000004.59.1, 00000000004.64.1, 00000000006.71.5, 00000000004.50.1, 00000000004.37.1, 00000000004.38.1] 2020-11-19T03:16:16.008 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:16.001+0000 7f256be6a0c0 0 bucket checkpoint complete 2020-11-19T03:16:16.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:16.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:16.140 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:16.185 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:16.186 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:16.443 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:16.439+0000 7ff0c2cd30c0 1 bucket sync caught up with source: 2020-11-19T03:16:16.443 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000002.50.5, , 00000000002.45.5, , 00000000002.36.5, , , 00000000002.67.5, , , ] 2020-11-19T03:16:16.443 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.50.5, , 00000000002.45.5, , 00000000002.36.5, , , 00000000002.67.5, , , ] 2020-11-19T03:16:16.444 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:16.439+0000 7ff0c2cd30c0 0 bucket checkpoint complete 2020-11-19T03:16:16.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:16.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:16.476 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:16.826 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:16.823+0000 7f77821980c0 1 bucket sync caught up with source: 2020-11-19T03:16:16.826 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000002.50.5, , 00000000002.45.5, , 00000000002.36.5, , , 00000000002.67.5, , , ] 2020-11-19T03:16:16.826 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.50.5, , 00000000002.45.5, , 00000000002.36.5, , , 00000000002.67.5, , , ] 2020-11-19T03:16:16.827 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:16.823+0000 7f77821980c0 0 bucket checkpoint complete 2020-11-19T03:16:16.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:16.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:16.859 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:16.947 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:16.947 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:17.213 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:17.207+0000 7f04fa5380c0 1 bucket sync caught up with source: 2020-11-19T03:16:17.229 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.52.5, 00000000002.39.1, 00000000004.46.5, 00000000002.54.1, 00000000004.44.5, 00000000002.58.1, 00000000002.88.1, 00000000005.99.5, 00000000002.52.1, 00000000002.40.1, 00000000002.40.1] 2020-11-19T03:16:17.230 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.52.5, 00000000002.39.1, 00000000004.46.5, 00000000002.54.1, 00000000004.44.5, 00000000002.58.1, 00000000002.88.1, 00000000005.99.5, 00000000002.52.1, 00000000002.40.1, 00000000002.40.1] 2020-11-19T03:16:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:17.207+0000 7f04fa5380c0 0 bucket checkpoint complete 2020-11-19T03:16:17.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:17.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:17.243 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:17.301 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:17.302 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:17.544 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:17.536+0000 7fa85e5990c0 1 bucket sync caught up with source: 2020-11-19T03:16:17.544 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.52.5, 00000000002.39.1, 00000000004.46.5, 00000000002.54.1, 00000000004.44.5, 00000000002.58.1, 00000000002.88.1, 00000000005.99.5, 00000000002.52.1, 00000000002.40.1, 00000000002.40.1] 2020-11-19T03:16:17.545 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.52.5, 00000000002.39.1, 00000000004.46.5, 00000000002.54.1, 00000000004.44.5, 00000000002.58.1, 00000000002.88.1, 00000000005.99.5, 00000000002.52.1, 00000000002.40.1, 00000000002.40.1] 2020-11-19T03:16:17.545 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:17.536+0000 7fa85e5990c0 0 bucket checkpoint complete 2020-11-19T03:16:17.579 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-60 zones={test-zone2, test-zone1} 2020-11-19T03:16:17.599 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:17.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:17.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:17.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:17.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:17.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:17.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:17.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:17.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:17.620 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:17.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:17.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:17.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:17.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:17.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:17.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:17.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:17.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:17.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:17.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:17.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:17.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:18.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:18.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:18.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:18.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:18.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:18.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:18.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:18.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:18.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:18.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:19.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:19.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:19.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:19.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:19.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:19.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:19.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:19.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:19.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:20.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:20.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-60 zones={test-zone2, test-zone1} 2020-11-19T03:16:20.200 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-60 zones={test-zone3, test-zone1} 2020-11-19T03:16:20.215 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:20.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:20.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:20.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:20.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:20.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:20.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:20.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:20.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:20.249 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:20.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:20.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:20.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:20.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:20.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:20.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:20.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:20.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:20.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:20.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:20.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:20.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:20.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:20.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:20.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:20.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:21.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:21.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:21.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:21.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:21.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:21.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:21.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:21.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:21.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:22.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:22.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:22.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:22.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:22.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:22.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:22.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:22.749 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-60 zones={test-zone3, test-zone1} 2020-11-19T03:16:22.796 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-60 zones={test-zone3, test-zone2} 2020-11-19T03:16:22.822 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:22.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:22.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:22.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:22.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:22.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:22.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:22.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:22.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:22.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:22.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:22.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:22.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:22.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:22.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:22.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:22.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:22.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:22.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:22.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:23.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:23.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:23.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:23.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:23.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:23.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:23.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:23.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:23.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:24.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:24.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:24.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:24.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:24.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:24.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:24.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:24.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:24.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:25.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:25.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:25.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:25.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:25.572 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-60 zones={test-zone3, test-zone2} 2020-11-19T03:16:25.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:25.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:25.574 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:25.624 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:25.624 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:25.866 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:25.856+0000 7f94022620c0 1 bucket sync caught up with source: 2020-11-19T03:16:25.866 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.43.5, 00000000004.42.1, 00000000006.91.5, 00000000004.69.1, 00000000006.77.5, 00000000004.39.1, 00000000004.44.1, 00000000006.32.5, 00000000004.39.1, 00000000004.154.1, 00000000004.41.1] 2020-11-19T03:16:25.866 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.43.5, 00000000004.42.1, 00000000006.91.5, 00000000004.69.1, 00000000006.77.5, 00000000004.39.1, 00000000004.44.1, 00000000006.32.5, 00000000004.39.1, 00000000004.154.1, 00000000004.41.1] 2020-11-19T03:16:25.867 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:25.856+0000 7f94022620c0 0 bucket checkpoint complete 2020-11-19T03:16:25.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:25.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:25.899 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:25.945 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:25.946 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:26.198 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:26.195+0000 7f4105fcb0c0 1 bucket sync caught up with source: 2020-11-19T03:16:26.198 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000006.43.5, 00000000004.42.1, 00000000006.91.5, 00000000004.69.1, 00000000006.77.5, 00000000004.39.1, 00000000004.44.1, 00000000006.32.5, 00000000004.39.1, 00000000004.154.1, 00000000004.41.1] 2020-11-19T03:16:26.198 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000006.43.5, 00000000004.42.1, 00000000006.91.5, 00000000004.69.1, 00000000006.77.5, 00000000004.39.1, 00000000004.44.1, 00000000006.32.5, 00000000004.39.1, 00000000004.154.1, 00000000004.41.1] 2020-11-19T03:16:26.198 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:26.195+0000 7f4105fcb0c0 0 bucket checkpoint complete 2020-11-19T03:16:26.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:26.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:26.226 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:26.283 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:26.284 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:26.515 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:26.508+0000 7fc68fe730c0 1 bucket sync caught up with source: 2020-11-19T03:16:26.516 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.43.5, 00000000004.42.1, 00000000006.91.5, 00000000004.69.1, 00000000006.77.5, 00000000004.39.1, 00000000004.44.1, 00000000006.32.5, 00000000004.39.1, 00000000004.154.1, 00000000004.41.1] 2020-11-19T03:16:26.516 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.43.5, 00000000004.42.1, 00000000006.91.5, 00000000004.69.1, 00000000006.77.5, 00000000004.39.1, 00000000004.44.1, 00000000006.32.5, 00000000004.39.1, 00000000004.154.1, 00000000004.41.1] 2020-11-19T03:16:26.516 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:26.508+0000 7fc68fe730c0 0 bucket checkpoint complete 2020-11-19T03:16:26.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:26.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:26.555 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:26.606 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:26.607 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:26.876 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:26.871+0000 7fc77e0440c0 1 bucket sync caught up with source: 2020-11-19T03:16:26.877 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000002.39.5, , 00000000002.93.5, , 00000000002.79.5, , , 00000000002.28.5, , , ] 2020-11-19T03:16:26.877 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.39.5, , 00000000002.93.5, , 00000000002.79.5, , , 00000000002.28.5, , , ] 2020-11-19T03:16:26.877 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:26.871+0000 7fc77e0440c0 0 bucket checkpoint complete 2020-11-19T03:16:26.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:26.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:26.913 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:26.967 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:26.967 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:27.187 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:27.183+0000 7f90743d60c0 1 bucket sync caught up with source: 2020-11-19T03:16:27.188 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000002.39.5, , 00000000002.93.5, , 00000000002.79.5, , , 00000000002.28.5, , , ] 2020-11-19T03:16:27.188 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.39.5, , 00000000002.93.5, , 00000000002.79.5, , , 00000000002.28.5, , , ] 2020-11-19T03:16:27.188 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:27.183+0000 7f90743d60c0 0 bucket checkpoint complete 2020-11-19T03:16:27.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:27.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:27.219 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:27.268 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:27.268 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:27.534 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:27.531+0000 7f9d235a20c0 1 bucket sync caught up with source: 2020-11-19T03:16:27.535 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000002.39.5, , 00000000002.103.5, , 00000000002.75.5, , , 00000000002.25.5, , , ] 2020-11-19T03:16:27.535 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000002.39.5, , 00000000002.103.5, , 00000000002.75.5, , , 00000000002.25.5, , , ] 2020-11-19T03:16:27.535 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:27.531+0000 7f9d235a20c0 0 bucket checkpoint complete 2020-11-19T03:16:27.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:27.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:27.574 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:27.621 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:27.622 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:27.888 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:27.880+0000 7fc0392dd0c0 1 bucket sync caught up with source: 2020-11-19T03:16:28.264 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.39.5, , 00000000002.103.5, , 00000000002.75.5, , , 00000000002.25.5, , , ] 2020-11-19T03:16:28.264 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.39.5, , 00000000002.103.5, , 00000000002.75.5, , , 00000000002.25.5, , , ] 2020-11-19T03:16:28.264 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:27.880+0000 7fc0392dd0c0 0 bucket checkpoint complete 2020-11-19T03:16:28.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-61 zones={test-zone2, test-zone1} 2020-11-19T03:16:28.278 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:28.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:28.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:28.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:28.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:28.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:28.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:28.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:28.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:28.291 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:28.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:28.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:28.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:28.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:28.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:28.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:28.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:28.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:28.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:28.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:28.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:28.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:28.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:28.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:28.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:29.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:29.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:29.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:29.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:29.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:29.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:29.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:29.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:29.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:30.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:30.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:30.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:30.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:30.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:30.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:30.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:30.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:30.982 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-61 zones={test-zone2, test-zone1} 2020-11-19T03:16:30.983 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-61 zones={test-zone3, test-zone1} 2020-11-19T03:16:31.002 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:31.021 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:31.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:31.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:31.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:31.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:31.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:31.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:31.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:31.028 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:31.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:31.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:31.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:31.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:31.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:31.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:31.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:31.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:31.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:31.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:31.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:31.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:31.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:31.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:31.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:31.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:31.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:32.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:32.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:32.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:32.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:32.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:32.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:32.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:32.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:33.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:33.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:33.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:33.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:33.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:33.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:33.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:34.042 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-61 zones={test-zone3, test-zone1} 2020-11-19T03:16:34.043 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-61 zones={test-zone3, test-zone2} 2020-11-19T03:16:34.061 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:34.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:34.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:34.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:34.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:34.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:34.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:34.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:34.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:34.074 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:34.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:34.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:34.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:34.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:34.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:16:34.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:16:34.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:16:34.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:16:34.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:34.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:34.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:34.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:34.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:34.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:34.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:34.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:34.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:35.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:35.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:35.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:35.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:35.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:35.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:16:35.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:35.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:35.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:16:35.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:36.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:36.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:16:36.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:36.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:36.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:16:36.569 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-61 zones={test-zone3, test-zone2} 2020-11-19T03:16:36.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:36.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:36.570 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:36.614 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:36.614 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:36.892 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:16:36.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:36.893 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-19T03:16:36.893 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:28.618911Z" 2020-11-19T03:16:36.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:36.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:36.894 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-19T03:16:36.894 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:58.758305Z" 2020-11-19T03:16:36.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:36.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:36.895 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-19T03:16:36.895 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:49.323602Z" 2020-11-19T03:16:36.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:36.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:36.895 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000012594", 2020-11-19T03:16:36.896 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:58.791936Z" 2020-11-19T03:16:36.896 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:36.896 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:16:36.897 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:16:36.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:36.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:36.899 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:36.943 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:36.943 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:37.238 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:16:37.238 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:16:37.239 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:16:37.239 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:16:37.239 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:16:37.239 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16121403893923359777 2020-11-19T03:16:37.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:37.240 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:16:37.240 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:37.240 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:16:37.240 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:37.241 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:37.241 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-19T03:16:37.241 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:37.241 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:37.241 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:37.242 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:28.618911Z" 2020-11-19T03:16:37.242 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:37.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:37.242 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:37.242 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:16:37.243 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:37.243 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:37.243 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-19T03:16:37.243 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:37.243 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:37.244 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:37.244 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:58.758305Z" 2020-11-19T03:16:37.244 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:37.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:37.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-19T03:16:37.246 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:37.246 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:37.246 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:37.246 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:49.323602Z" 2020-11-19T03:16:37.246 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:37.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:37.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:37.247 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:16:37.247 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:37.247 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:37.248 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000012594", 2020-11-19T03:16:37.248 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:37.248 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:37.248 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:37.248 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:58.791936Z" 2020-11-19T03:16:37.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:37.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:37.249 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:16:37.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:37.249 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:16:37.250 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:16:37.250 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:16:37.250 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:37.250 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:16:37.251 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:16:37.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:37.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:37.253 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:37.294 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:37.576 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:16:37.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:37.577 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-19T03:16:37.577 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:28.618911Z" 2020-11-19T03:16:37.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:37.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:37.578 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-19T03:16:37.578 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:58.758305Z" 2020-11-19T03:16:37.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:37.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:37.579 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-19T03:16:37.579 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:49.323602Z" 2020-11-19T03:16:37.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:37.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:37.580 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000012594", 2020-11-19T03:16:37.580 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:58.791936Z" 2020-11-19T03:16:37.580 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:37.580 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:16:37.581 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:16:37.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:37.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:37.583 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:37.665 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:37.665 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:37.927 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:16:37.928 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:16:37.928 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:16:37.928 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:16:37.928 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:16:37.929 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7000157538207389965 2020-11-19T03:16:37.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:37.929 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:16:37.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:37.929 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:16:37.930 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:37.930 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:37.930 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-19T03:16:37.930 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:37.931 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:37.931 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:37.931 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:28.618911Z" 2020-11-19T03:16:37.931 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:37.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:37.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:37.932 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:16:37.932 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:37.932 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:37.932 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-19T03:16:37.933 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:37.933 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:37.933 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:37.933 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:58.758305Z" 2020-11-19T03:16:37.933 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:37.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:37.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:37.934 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:16:37.934 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:37.934 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:37.935 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-19T03:16:37.935 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:37.935 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:37.935 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:37.936 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:49.323602Z" 2020-11-19T03:16:37.936 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:37.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:37.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:37.936 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:16:37.937 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:37.937 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:37.937 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000012594", 2020-11-19T03:16:37.937 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:37.937 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:37.938 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:37.938 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:58.791936Z" 2020-11-19T03:16:37.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:37.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:37.938 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:16:37.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:37.939 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:16:37.939 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:16:37.939 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:16:37.939 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:37.940 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:16:37.941 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:16:37.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:37.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:37.942 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:38.038 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:38.038 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:38.313 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:16:38.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:38.314 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-19T03:16:38.314 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:28.618911Z" 2020-11-19T03:16:38.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:38.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:38.315 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-19T03:16:38.316 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:58.758305Z" 2020-11-19T03:16:38.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:38.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:38.317 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-19T03:16:38.317 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:49.323602Z" 2020-11-19T03:16:38.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:38.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:38.318 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000012594", 2020-11-19T03:16:38.318 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:58.791936Z" 2020-11-19T03:16:38.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:38.319 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:16:38.321 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:16:38.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:38.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:38.323 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:38.366 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:38.366 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:38.632 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:16:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:16:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:16:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:16:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:16:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 12326036592617550149 2020-11-19T03:16:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:16:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:16:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-19T03:16:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:28.618911Z" 2020-11-19T03:16:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:16:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-19T03:16:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:58.758305Z" 2020-11-19T03:16:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:16:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-19T03:16:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:49.323602Z" 2020-11-19T03:16:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:16:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000012594", 2020-11-19T03:16:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:58.791936Z" 2020-11-19T03:16:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:16:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:16:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:16:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:16:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:38.645 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:16:38.646 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:16:38.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:38.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:38.647 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:38.729 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:38.730 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:39.013 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:16:39.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:39.096 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006885", 2020-11-19T03:16:39.097 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.496151Z" 2020-11-19T03:16:39.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:39.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:39.100 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002770", 2020-11-19T03:16:39.100 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:15:49.202535Z" 2020-11-19T03:16:39.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:39.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:39.101 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003964", 2020-11-19T03:16:39.101 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:15:49.207260Z" 2020-11-19T03:16:39.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:39.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:39.102 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006751", 2020-11-19T03:16:39.102 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:15:49.211609Z" 2020-11-19T03:16:39.102 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:39.102 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:16:39.103 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:16:39.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:39.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:39.105 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:39.152 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:39.154 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:39.439 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:16:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:16:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:16:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:16:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:16:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7306459910885960034 2020-11-19T03:16:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:16:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:16:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006885", 2020-11-19T03:16:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.496151Z" 2020-11-19T03:16:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:16:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002770", 2020-11-19T03:16:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:49.202535Z" 2020-11-19T03:16:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:16:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003964", 2020-11-19T03:16:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:49.207260Z" 2020-11-19T03:16:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:16:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006751", 2020-11-19T03:16:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:49.211609Z" 2020-11-19T03:16:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:16:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:16:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:16:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:16:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:39.457 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:16:39.459 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:16:39.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:39.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:39.461 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:39.503 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:39.503 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:39.781 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:16:39.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:39.782 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006885", 2020-11-19T03:16:39.782 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:14:38.496151Z" 2020-11-19T03:16:39.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:39.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:39.783 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002770", 2020-11-19T03:16:39.783 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:15:49.202535Z" 2020-11-19T03:16:39.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:39.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:39.784 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003964", 2020-11-19T03:16:39.784 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:15:49.207260Z" 2020-11-19T03:16:39.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:39.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:39.784 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006751", 2020-11-19T03:16:39.785 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:15:49.211609Z" 2020-11-19T03:16:39.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:39.785 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:16:39.786 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:16:39.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:39.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:39.788 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:39.833 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:39.834 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:40.102 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:16:40.102 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:16:40.103 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:16:40.103 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:16:40.103 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:16:40.103 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 597164135494128594 2020-11-19T03:16:40.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.104 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:16:40.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.104 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:16:40.104 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:40.104 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:40.105 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006885", 2020-11-19T03:16:40.105 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:40.105 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:40.105 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:40.106 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:14:38.496151Z" 2020-11-19T03:16:40.106 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.106 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:16:40.107 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:40.107 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:40.107 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000002770", 2020-11-19T03:16:40.107 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:40.107 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:40.108 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:40.108 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:49.202535Z" 2020-11-19T03:16:40.108 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.109 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:16:40.109 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:40.109 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:40.109 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003964", 2020-11-19T03:16:40.109 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:40.110 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:40.110 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:40.110 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:49.207260Z" 2020-11-19T03:16:40.110 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.111 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:16:40.111 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:40.111 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:40.111 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006751", 2020-11-19T03:16:40.112 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:40.112 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:40.112 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:40.112 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:49.211609Z" 2020-11-19T03:16:40.112 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.113 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.113 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:16:40.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.113 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:16:40.113 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:16:40.114 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:16:40.114 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.114 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:16:40.115 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:16:40.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:40.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:40.117 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:40.194 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:40.195 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:40.462 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:16:40.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.463 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007550", 2020-11-19T03:16:40.463 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:44.929189Z" 2020-11-19T03:16:40.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.465 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003570", 2020-11-19T03:16:40.466 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:16:06.947886Z" 2020-11-19T03:16:40.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.467 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000004496", 2020-11-19T03:16:40.468 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:16:06.952337Z" 2020-11-19T03:16:40.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.469 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007416", 2020-11-19T03:16:40.470 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:16:06.956347Z" 2020-11-19T03:16:40.470 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.472 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:16:40.474 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:16:40.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:40.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:40.478 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:40.561 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:40.562 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:40.839 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:16:40.840 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:16:40.840 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:16:40.840 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:16:40.840 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:16:40.841 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 2448524765768921960 2020-11-19T03:16:40.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.841 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:16:40.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.842 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:16:40.842 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:40.842 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:40.842 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007550", 2020-11-19T03:16:40.842 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:40.843 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:40.843 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:40.843 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:15:44.929189Z" 2020-11-19T03:16:40.844 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.844 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:16:40.845 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:40.845 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:40.845 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003570", 2020-11-19T03:16:40.845 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:40.845 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:40.846 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:40.846 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:16:06.947886Z" 2020-11-19T03:16:40.846 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.847 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:16:40.847 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:40.847 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:40.847 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000004496", 2020-11-19T03:16:40.848 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:40.848 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:40.848 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:40.848 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:16:06.952337Z" 2020-11-19T03:16:40.849 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:40.849 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:16:40.849 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:40.850 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:16:40.850 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007416", 2020-11-19T03:16:40.850 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:40.850 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:40.850 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:40.851 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:16:06.956347Z" 2020-11-19T03:16:40.851 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.851 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.851 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:16:40.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:40.852 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:16:40.852 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:16:40.852 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:16:40.852 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:40.853 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:16:40.854 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:16:40.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:40.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:40.856 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:40.938 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:40.938 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:41.194 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:16:41.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:41.195 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007550", 2020-11-19T03:16:41.195 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:15:44.929189Z" 2020-11-19T03:16:41.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:41.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:41.195 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000003570", 2020-11-19T03:16:41.196 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:16:06.947886Z" 2020-11-19T03:16:41.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:41.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:41.196 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000004496", 2020-11-19T03:16:41.197 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:16:06.952337Z" 2020-11-19T03:16:41.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:41.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:41.197 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000007416", 2020-11-19T03:16:41.197 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:16:06.956347Z" 2020-11-19T03:16:41.198 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:41.198 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:16:41.199 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:16:41.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:41.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:41.200 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:41.252 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:41.252 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:41.532 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:16:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:16:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:16:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:16:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:16:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16483361435957904979 2020-11-19T03:16:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:16:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:16:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000007550", 2020-11-19T03:16:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:15:44.929189Z" 2020-11-19T03:16:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:16:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:41.541 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003570", 2020-11-19T03:16:41.541 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:41.541 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:41.541 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:41.542 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:16:06.947886Z" 2020-11-19T03:16:41.542 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:41.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:41.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:41.542 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:16:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000004496", 2020-11-19T03:16:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:16:06.952337Z" 2020-11-19T03:16:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:16:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:16:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000007416", 2020-11-19T03:16:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:16:06.956347Z" 2020-11-19T03:16:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:16:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:16:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:16:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:16:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:41.548 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:16:41.548 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:16:41.549 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2020-11-19T03:16:41.551 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-62 2020-11-19T03:16:41.599 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-63 2020-11-19T03:16:41.672 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-64 2020-11-19T03:16:41.765 INFO:tasks.rgw.c1.client.1.smithi086.stdout:2020-11-19T03:16:41.758+0000 7f97817fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:16:41.766 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2020-11-19T03:16:41.760+0000 7ff428ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:16:42.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:42.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:42.375 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:42.414 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:42.414 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:42.676 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:16:42.703 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:16:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755801.715459_864.1", 2020-11-19T03:16:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:16:41.715459Z" 2020-11-19T03:16:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:16:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:16:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:16:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:16:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:16:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:16:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:42.706 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:16:42.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:16:42.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:42.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:42.709 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:42.750 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:42.751 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:42.980 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:16:42.981 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:16:42.981 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:16:42.982 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:16:42.982 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:16:42.982 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:16:42.983 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:16:42.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:42.984 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:16:42.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:42.984 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:16:42.985 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:42.986 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:16:42.986 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755801.715459_864.1", 2020-11-19T03:16:42.987 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:42.988 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:16:42.988 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:42.988 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:16:41.715459Z", 2020-11-19T03:16:42.989 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:16:42.989 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:42.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:42.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:42.991 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:16:42.991 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:42.991 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:16:42.992 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:16:42.992 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:42.993 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:42.993 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:42.993 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:16:42.994 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:16:42.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:42.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:42.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:42.995 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:16:42.996 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:42.996 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:16:42.997 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:16:42.998 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:42.998 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:42.999 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:42.999 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:16:42.999 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:16:43.000 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:43.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:43.001 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:16:43.001 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:43.002 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:16:43.002 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:16:43.003 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:43.003 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:43.003 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:43.004 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:16:43.004 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:16:43.005 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.005 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.005 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:16:43.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:43.006 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:16:43.006 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:16:43.007 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:16:43.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.007 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:16:43.009 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:16:43.009 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:16:43.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:43.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:43.011 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:43.054 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:43.054 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:43.326 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:16:43.327 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:16:43.327 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:16:43.327 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:16:43.327 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:16:43.328 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:16:43.328 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:16:43.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:43.328 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:16:43.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:43.329 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:16:43.329 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:43.329 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:16:43.329 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755801.715459_864.1", 2020-11-19T03:16:43.330 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:43.330 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:16:43.330 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:43.330 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:16:41.715459Z", 2020-11-19T03:16:43.330 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:16:43.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:43.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:43.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:43.331 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:16:43.331 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:43.332 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:16:43.332 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:16:43.332 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:43.332 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:43.332 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:43.333 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:16:43.333 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:16:43.333 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:43.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:43.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:43.334 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:16:43.334 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:43.334 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:16:43.334 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:16:43.334 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:43.335 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:43.335 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:43.335 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:16:43.335 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:16:43.335 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:43.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:43.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:16:43.336 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:16:43.336 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:16:43.336 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:16:43.337 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:16:43.337 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:16:43.337 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:16:43.337 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:16:43.337 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:16:43.338 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:16:43.338 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:43.338 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:43.338 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:16:43.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:16:43.339 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:16:43.339 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:16:43.339 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:16:43.339 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:16:43.339 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:16:43.340 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:16:43.340 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:16:43.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:43.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:16:43.342 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:16:43.391 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:43.391 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:43.660 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:16:43.660 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:16:43.660 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:16:43.661 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:16:43.661 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:16:43.661 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:16:43.661 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:16:43.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:43.662 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:16:43.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:43.662 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:16:43.662 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:43.662 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:16:43.663 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755801.715459_864.1", 2020-11-19T03:16:43.663 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:43.663 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:16:43.663 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:43.664 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:16:41.715459Z", 2020-11-19T03:16:43.665 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:16:43.665 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:43.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:43.666 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:16:43.666 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:43.666 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:16:43.666 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:16:43.666 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:43.667 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:43.667 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:43.667 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:16:43.667 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:16:43.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:43.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:43.668 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:16:43.668 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:43.668 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:16:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:16:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:16:43.670 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:16:43.670 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:43.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:16:43.670 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:16:43.671 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:16:43.671 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:16:43.671 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:16:43.671 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:16:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:16:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:16:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:16:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:16:43.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:16:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:16:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:16:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:16:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:16:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:16:43.674 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:16:43.676 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:16:43.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:43.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:43.677 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:43.764 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:43.764 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:44.027 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:44.020+0000 7f71edcf60c0 1 bucket sync caught up with source: 2020-11-19T03:16:44.027 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.43.5, , 00000000001.81.5, , 00000000001.65.5, , , 00000000001.50.5, , , ] 2020-11-19T03:16:44.028 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.43.5, , 00000000001.81.5, , 00000000001.65.5, , , 00000000001.50.5, , , ] 2020-11-19T03:16:44.028 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:44.020+0000 7f71edcf60c0 0 bucket checkpoint complete 2020-11-19T03:16:44.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:44.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:44.063 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:44.358 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:44.354+0000 7f710fd560c0 1 bucket sync caught up with source: 2020-11-19T03:16:44.358 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.43.5, , 00000000001.81.5, , 00000000001.65.5, , , 00000000001.50.5, , , ] 2020-11-19T03:16:44.358 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.43.5, , 00000000001.81.5, , 00000000001.65.5, , , 00000000001.50.5, , , ] 2020-11-19T03:16:44.359 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:44.354+0000 7f710fd560c0 0 bucket checkpoint complete 2020-11-19T03:16:44.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:44.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:44.388 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:44.436 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:44.436 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:44.684 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:44.676+0000 7f4316e090c0 1 bucket sync caught up with source: 2020-11-19T03:16:44.685 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.43.5, , 00000000001.81.5, , 00000000001.65.5, , , 00000000001.50.5, , , ] 2020-11-19T03:16:44.685 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.43.5, , 00000000001.81.5, , 00000000001.65.5, , , 00000000001.50.5, , , ] 2020-11-19T03:16:44.685 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:44.676+0000 7f4316e090c0 0 bucket checkpoint complete 2020-11-19T03:16:44.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:44.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:44.810 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:44.861 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:44.861 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:45.115 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:45.110+0000 7ff7d5ad20c0 1 bucket sync caught up with source: 2020-11-19T03:16:45.115 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.46.5, , 00000000001.84.5, , 00000000001.54.5, , , 00000000001.51.5, , , ] 2020-11-19T03:16:45.115 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.46.5, , 00000000001.84.5, , 00000000001.54.5, , , 00000000001.51.5, , , ] 2020-11-19T03:16:45.116 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:45.110+0000 7ff7d5ad20c0 0 bucket checkpoint complete 2020-11-19T03:16:45.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:45.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:45.148 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:45.198 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:45.200 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:45.441 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:45.434+0000 7f2e1fc4a0c0 1 bucket sync caught up with source: 2020-11-19T03:16:45.441 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.46.5, , 00000000001.84.5, , 00000000001.54.5, , , 00000000001.51.5, , , ] 2020-11-19T03:16:45.441 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.46.5, , 00000000001.84.5, , 00000000001.54.5, , , 00000000001.51.5, , , ] 2020-11-19T03:16:45.442 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:45.434+0000 7f2e1fc4a0c0 0 bucket checkpoint complete 2020-11-19T03:16:45.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:45.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:45.473 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:45.516 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:45.517 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:45.786 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:45.782+0000 7fb7a481e0c0 1 bucket sync caught up with source: 2020-11-19T03:16:45.786 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.44.5, , 00000000001.83.5, , 00000000001.53.5, , , 00000000001.53.5, , , ] 2020-11-19T03:16:45.787 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.44.5, , 00000000001.83.5, , 00000000001.53.5, , , 00000000001.53.5, , , ] 2020-11-19T03:16:45.787 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:45.782+0000 7fb7a481e0c0 0 bucket checkpoint complete 2020-11-19T03:16:45.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:45.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:45.818 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:45.876 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:45.878 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:46.127 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:46.120+0000 7fbb570500c0 1 bucket sync caught up with source: 2020-11-19T03:16:46.128 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.44.5, , 00000000001.83.5, , 00000000001.53.5, , , 00000000001.53.5, , , ] 2020-11-19T03:16:46.128 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.44.5, , 00000000001.83.5, , 00000000001.53.5, , , 00000000001.53.5, , , ] 2020-11-19T03:16:46.128 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:46.120+0000 7fbb570500c0 0 bucket checkpoint complete 2020-11-19T03:16:46.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-62 zones={test-zone2, test-zone1} 2020-11-19T03:16:46.176 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:46.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:46.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:46.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:46.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:46.184 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:46.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:46.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:46.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:46.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:46.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:46.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:46.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:46.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:46.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:46.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:46.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:46.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:47.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:47.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:47.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:47.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:47.444 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-62 zones={test-zone2, test-zone1} 2020-11-19T03:16:47.445 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-62 zones={test-zone3, test-zone1} 2020-11-19T03:16:47.456 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:47.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:47.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:47.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:47.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:47.465 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:47.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:47.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:47.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:47.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:47.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:47.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:47.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:47.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:47.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:47.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:48.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:48.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:48.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:48.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:48.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:48.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:48.660 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-62 zones={test-zone3, test-zone1} 2020-11-19T03:16:48.661 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-62 zones={test-zone3, test-zone2} 2020-11-19T03:16:48.672 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:48.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:48.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:48.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:48.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:48.681 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:48.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:48.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:48.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:48.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:48.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:49.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:49.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:49.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:49.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:49.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:49.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:49.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:49.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:49.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:49.985 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-62 zones={test-zone3, test-zone2} 2020-11-19T03:16:49.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:49.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:49.987 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:50.036 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:50.036 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:50.291 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:50.284+0000 7f3363e940c0 1 bucket sync caught up with source: 2020-11-19T03:16:50.291 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.42.5, , 00000000001.94.5, , 00000000001.43.5, , , 00000000001.45.5, , , ] 2020-11-19T03:16:50.292 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.42.5, , 00000000001.94.5, , 00000000001.43.5, , , 00000000001.45.5, , , ] 2020-11-19T03:16:50.292 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:50.284+0000 7f3363e940c0 0 bucket checkpoint complete 2020-11-19T03:16:50.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:50.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:50.324 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:50.632 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:50.626+0000 7f45bb60e0c0 1 bucket sync caught up with source: 2020-11-19T03:16:50.632 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.42.5, , 00000000001.94.5, , 00000000001.43.5, , , 00000000001.45.5, , , ] 2020-11-19T03:16:50.633 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.42.5, , 00000000001.94.5, , 00000000001.43.5, , , 00000000001.45.5, , , ] 2020-11-19T03:16:50.633 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:50.626+0000 7f45bb60e0c0 0 bucket checkpoint complete 2020-11-19T03:16:50.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:50.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:50.664 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:50.715 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:50.715 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:50.967 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:50.960+0000 7f4c7146f0c0 1 bucket sync caught up with source: 2020-11-19T03:16:50.968 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.42.5, , 00000000001.94.5, , 00000000001.43.5, , , 00000000001.45.5, , , ] 2020-11-19T03:16:50.968 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.42.5, , 00000000001.94.5, , 00000000001.43.5, , , 00000000001.45.5, , , ] 2020-11-19T03:16:50.968 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:50.960+0000 7f4c7146f0c0 0 bucket checkpoint complete 2020-11-19T03:16:51.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:51.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:51.088 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:51.130 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:51.130 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:51.389 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:51.382+0000 7fa78eaa40c0 1 bucket sync caught up with source: 2020-11-19T03:16:51.389 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.44.5, , 00000000001.97.5, , 00000000001.34.5, , , 00000000001.44.5, , , ] 2020-11-19T03:16:51.390 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.44.5, , 00000000001.97.5, , 00000000001.34.5, , , 00000000001.44.5, , , ] 2020-11-19T03:16:51.390 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:51.382+0000 7fa78eaa40c0 0 bucket checkpoint complete 2020-11-19T03:16:51.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:51.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:51.423 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:51.512 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:51.513 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:51.772 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:51.766+0000 7f0bb40bf0c0 1 bucket sync caught up with source: 2020-11-19T03:16:51.773 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.44.5, , 00000000001.97.5, , 00000000001.34.5, , , 00000000001.44.5, , , ] 2020-11-19T03:16:51.773 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.44.5, , 00000000001.97.5, , 00000000001.34.5, , , 00000000001.44.5, , , ] 2020-11-19T03:16:51.773 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:51.766+0000 7f0bb40bf0c0 0 bucket checkpoint complete 2020-11-19T03:16:51.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:51.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:51.805 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:52.102+0000 7f0abc4550c0 1 bucket sync caught up with source: 2020-11-19T03:16:52.108 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.43.5, , 00000000001.108.5, , 00000000001.40.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:52.108 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.43.5, , 00000000001.108.5, , 00000000001.40.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:52.102+0000 7f0abc4550c0 0 bucket checkpoint complete 2020-11-19T03:16:52.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:52.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:52.138 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:52.188 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:52.188 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:52.461 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:52.448+0000 7f63f82a60c0 1 bucket sync caught up with source: 2020-11-19T03:16:52.462 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.43.5, , 00000000001.108.5, , 00000000001.40.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:52.462 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.43.5, , 00000000001.108.5, , 00000000001.40.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:52.463 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:52.448+0000 7f63f82a60c0 0 bucket checkpoint complete 2020-11-19T03:16:52.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-63 zones={test-zone2, test-zone1} 2020-11-19T03:16:52.510 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:52.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:52.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:52.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:52.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:52.527 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:52.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:52.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:52.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:52.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:52.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:52.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:52.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:52.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:52.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:53.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:53.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:53.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:53.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:53.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:53.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:53.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:53.778 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-63 zones={test-zone2, test-zone1} 2020-11-19T03:16:53.779 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-63 zones={test-zone3, test-zone1} 2020-11-19T03:16:53.790 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:53.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:53.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:53.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:53.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:53.799 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:53.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:53.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:53.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:53.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:53.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:53.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:54.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:54.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:54.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:54.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:54.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:54.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:54.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:54.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:54.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:55.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:55.176 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-63 zones={test-zone3, test-zone1} 2020-11-19T03:16:55.177 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-63 zones={test-zone3, test-zone2} 2020-11-19T03:16:55.190 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:55.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:55.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:55.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:55.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:55.200 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:55.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:55.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:55.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:55.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:55.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:55.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:55.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:55.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:55.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:55.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:55.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:55.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:56.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:56.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:56.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:56.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:56.466 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-63 zones={test-zone3, test-zone2} 2020-11-19T03:16:56.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:56.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:56.468 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:56.518 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:56.519 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:56.785 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:56.776+0000 7f1348d790c0 1 bucket sync caught up with source: 2020-11-19T03:16:56.786 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.47.5, , 00000000001.83.5, , 00000000001.35.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:56.786 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.47.5, , 00000000001.83.5, , 00000000001.35.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:56.786 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:56.776+0000 7f1348d790c0 0 bucket checkpoint complete 2020-11-19T03:16:56.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:56.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:56.819 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:57.115 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:57.110+0000 7f4df319e0c0 1 bucket sync caught up with source: 2020-11-19T03:16:57.116 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.47.5, , 00000000001.83.5, , 00000000001.35.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:57.116 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.47.5, , 00000000001.83.5, , 00000000001.35.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:57.117 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:57.110+0000 7f4df319e0c0 0 bucket checkpoint complete 2020-11-19T03:16:57.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:57.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:57.145 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:57.187 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:57.188 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:57.416 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:57.408+0000 7fc5888460c0 1 bucket sync caught up with source: 2020-11-19T03:16:57.418 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.47.5, , 00000000001.83.5, , 00000000001.35.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:57.418 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.47.5, , 00000000001.83.5, , 00000000001.35.5, , , 00000000001.46.5, , , ] 2020-11-19T03:16:57.419 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:57.408+0000 7fc5888460c0 0 bucket checkpoint complete 2020-11-19T03:16:57.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:57.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:57.548 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:57.595 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:57.595 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:57.873 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:57.870+0000 7f0528d420c0 1 bucket sync caught up with source: 2020-11-19T03:16:57.874 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.52.5, , 00000000001.86.5, , 00000000001.31.5, , , 00000000001.42.5, , , ] 2020-11-19T03:16:57.874 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.52.5, , 00000000001.86.5, , 00000000001.31.5, , , 00000000001.42.5, , , ] 2020-11-19T03:16:57.874 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:57.870+0000 7f0528d420c0 0 bucket checkpoint complete 2020-11-19T03:16:57.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:57.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:57.909 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:57.959 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:57.959 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:58.222 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:58.218+0000 7f1d0b8c80c0 1 bucket sync caught up with source: 2020-11-19T03:16:58.223 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.52.5, , 00000000001.86.5, , 00000000001.31.5, , , 00000000001.42.5, , , ] 2020-11-19T03:16:58.223 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.52.5, , 00000000001.86.5, , 00000000001.31.5, , , 00000000001.42.5, , , ] 2020-11-19T03:16:58.223 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:58.218+0000 7f1d0b8c80c0 0 bucket checkpoint complete 2020-11-19T03:16:58.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:58.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:58.256 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:58.345 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:58.345 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:58.606 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:58.602+0000 7fa0ce8730c0 1 bucket sync caught up with source: 2020-11-19T03:16:58.606 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000001.46.5, , 00000000001.85.5, , 00000000001.31.5, , , 00000000001.47.5, , , ] 2020-11-19T03:16:58.606 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000001.46.5, , 00000000001.85.5, , 00000000001.31.5, , , 00000000001.47.5, , , ] 2020-11-19T03:16:58.607 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:16:58.602+0000 7fa0ce8730c0 0 bucket checkpoint complete 2020-11-19T03:16:58.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:58.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:16:58.639 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:16:58.687 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:16:58.687 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:16:58.956 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:58.948+0000 7ff25cdca0c0 1 bucket sync caught up with source: 2020-11-19T03:16:58.956 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.46.5, , 00000000001.85.5, , 00000000001.31.5, , , 00000000001.47.5, , , ] 2020-11-19T03:16:58.956 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.46.5, , 00000000001.85.5, , 00000000001.31.5, , , 00000000001.47.5, , , ] 2020-11-19T03:16:58.956 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:16:58.948+0000 7ff25cdca0c0 0 bucket checkpoint complete 2020-11-19T03:16:58.990 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-64 zones={test-zone2, test-zone1} 2020-11-19T03:16:59.003 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:16:59.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:59.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:59.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:59.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:59.015 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:16:59.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:16:59.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:16:59.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:16:59.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:16:59.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:59.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:59.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:16:59.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:59.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:59.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:16:59.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:59.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:59.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:16:59.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:16:59.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:00.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:00.241 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-64 zones={test-zone2, test-zone1} 2020-11-19T03:17:00.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-64 zones={test-zone3, test-zone1} 2020-11-19T03:17:00.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:17:00.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:00.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:00.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:00.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:00.303 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:17:00.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:00.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:00.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:00.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:00.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:00.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:00.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:00.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:00.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:00.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:00.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:01.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:01.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:01.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:01.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:01.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:01.573 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-64 zones={test-zone3, test-zone1} 2020-11-19T03:17:01.574 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-64 zones={test-zone3, test-zone2} 2020-11-19T03:17:01.585 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:17:01.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:01.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:01.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:01.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:01.597 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:17:01.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:01.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:01.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:01.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:01.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:01.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:01.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:01.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:01.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:02.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:02.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:02.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:02.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:02.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:02.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:02.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:02.810 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-64 zones={test-zone3, test-zone2} 2020-11-19T03:17:02.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'atsuux-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:02.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'atsuux-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:02.812 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket atsuux-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:17:02.858 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:02.858 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:03.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'atsuux-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:03.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'atsuux-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:03.232 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket atsuux-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:17:03.321 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:03.322 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:03.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'atsuux-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:03.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'atsuux-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:03.654 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket atsuux-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:17:03.742 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:03.743 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:04.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:04.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:04.053 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:17:04.092 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:04.093 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:04.316 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:17:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:17:04.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:17:04.342 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755824.003321_900.1", 2020-11-19T03:17:04.342 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:04.003321Z" 2020-11-19T03:17:04.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:17:04.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:17:04.343 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:17:04.343 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:17:04.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:17:04.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:17:04.343 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:17:04.344 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:17:04.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:17:04.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:17:04.344 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:17:04.344 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:17:04.345 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:17:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:17:04.346 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:17:04.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:04.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:04.347 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:17:04.387 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:04.387 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:04.664 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:17:04.664 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:17:04.664 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:17:04.665 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:17:04.665 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:17:04.665 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:17:04.665 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:17:04.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:04.666 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:17:04.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:04.666 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:17:04.666 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:04.666 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:04.667 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755823.164886_890.1", 2020-11-19T03:17:04.667 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:04.667 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:17:04.667 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:04.668 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:03.164886Z", 2020-11-19T03:17:04.668 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:17:04.668 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:04.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:04.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:17:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:17:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:04.670 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:04.670 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:04.670 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:17:04.670 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:04.670 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:04.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:04.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:04.671 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:17:04.671 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:04.672 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:04.672 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:17:04.672 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:04.672 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:04.673 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:04.673 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:17:04.673 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:04.673 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:04.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:04.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:04.674 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:17:04.674 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:04.674 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:04.675 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:17:04.675 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:04.675 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:04.675 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:04.676 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:17:04.676 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:04.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:04.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:04.676 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:17:04.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:04.677 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:17:04.677 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:17:04.677 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:17:04.677 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:04.678 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:17:04.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755824.003321_900.1 target=1_1605755823.164886_890.1 2020-11-19T03:17:09.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:09.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:09.682 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:17:09.723 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:09.724 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:10.006 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:17:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:17:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:17:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:17:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:17:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:17:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:17:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:17:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:17:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755823.164886_890.1", 2020-11-19T03:17:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:17:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:03.164886Z", 2020-11-19T03:17:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:17:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:17:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:17:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:17:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:17:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:17:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:17:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:17:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:17:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:17:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:17:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:17:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:17:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:17:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:10.019 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:17:10.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605755824.003321_900.1 target=1_1605755823.164886_890.1 2020-11-19T03:17:15.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:15.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:15.024 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:17:15.066 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:15.066 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:15.342 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:17:15.387 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:17:15.387 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:17:15.388 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:17:15.388 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:17:15.388 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:17:15.388 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:17:15.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:15.389 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:17:15.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:15.389 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:17:15.389 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:15.389 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:15.390 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755824.003321_900.1", 2020-11-19T03:17:15.390 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:15.390 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:17:15.390 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:15.391 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:04.003321Z", 2020-11-19T03:17:15.391 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:17:15.391 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:15.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:15.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:15.392 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:17:15.392 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:15.392 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:15.392 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:17:15.393 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:15.393 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:15.393 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:15.393 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:17:15.393 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:15.394 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:15.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:15.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:15.394 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:17:15.394 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:15.395 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:15.395 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:17:15.395 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:15.395 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:15.395 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:15.396 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:17:15.396 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:15.396 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:15.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:15.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:15.396 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:17:15.397 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:15.397 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:15.397 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:17:15.397 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:15.397 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:15.398 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:15.398 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:17:15.398 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:15.398 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:15.398 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:15.399 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:17:15.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:15.399 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:17:15.399 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:17:15.399 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:17:15.400 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:15.400 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:17:15.400 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:17:15.401 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:17:15.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:15.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:15.403 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:17:15.446 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:15.447 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:15.724 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:17:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:17:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:17:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:17:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:17:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:17:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:17:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:17:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:17:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:17:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:17:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:17:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:17:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755824.003321_900.1", 2020-11-19T03:17:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:17:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:17:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:17:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:04.003321Z", 2020-11-19T03:17:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:17:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:17:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:17:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:17:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:17:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:17:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:17:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:17:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:17:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:17:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:17:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:17:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:17:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:17:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:17:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:17:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:17:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:17:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:17:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:17:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:17:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:17:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:17:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:17:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:17:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:17:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:17:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:17:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:17:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:17:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:17:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:17:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:17:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:17:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:17:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:17:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:17:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:17:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:17:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:17:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:17:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:17:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:17:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:17:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:17:15.738 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:17:15.739 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:17:15.739 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:17:15.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:15.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:17:15.741 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:17:15.783 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:15.783 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:16.156 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:17:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:17:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:17:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:17:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:17:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:17:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:17:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:17:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:17:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755824.003321_900.1", 2020-11-19T03:17:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:17:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:04.003321Z", 2020-11-19T03:17:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:17:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:17:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:17:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:17:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:17:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:17:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:17:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:17:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:17:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:17:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:17:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:17:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:17:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:17:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:17:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:17:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:17:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:17:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:17:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:17:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:17:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:17:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:17:16.169 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:17:16.170 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:17:16.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'atsuux-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:16.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'atsuux-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:16.813 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket atsuux-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:17:16.868 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:17.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'atsuux-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:17.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'atsuux-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:17.198 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket atsuux-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:17:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:17.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'atsuux-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:17.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'atsuux-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:17:17.597 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket atsuux-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:17:17.684 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:17.686 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:18.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:18.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:18.035 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:18.084 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:18.085 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:18.317 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:18.308+0000 7fd8587b80c0 1 waiting for incremental sync to catch up: 2020-11-19T03:17:18.317 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:17:18.318 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.53.1, 00000000004.48.1, 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, 00000000004.40.1, 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, 00000000004.64.1, 00000000004.53.1] 2020-11-19T03:17:23.325 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:23.316+0000 7fd8587b80c0 1 waiting for incremental sync to catch up: 2020-11-19T03:17:23.326 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.53.1, , 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, , 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, , 00000000004.53.1] 2020-11-19T03:17:23.328 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.53.1, 00000000004.48.1, 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, 00000000004.40.1, 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, 00000000004.64.1, 00000000004.53.1] 2020-11-19T03:17:28.329 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:28.320+0000 7fd8587b80c0 1 waiting for incremental sync to catch up: 2020-11-19T03:17:28.329 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.53.1, , 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, , 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, , 00000000004.53.1] 2020-11-19T03:17:28.329 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.53.1, 00000000004.48.1, 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, 00000000004.40.1, 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, 00000000004.64.1, 00000000004.53.1] 2020-11-19T03:17:33.333 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:33.323+0000 7fd8587b80c0 1 bucket sync caught up with source: 2020-11-19T03:17:33.333 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.53.1, 00000000004.48.1, 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, 00000000004.40.1, 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, 00000000004.64.1, 00000000004.53.1] 2020-11-19T03:17:33.333 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.53.1, 00000000004.48.1, 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, 00000000004.40.1, 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, 00000000004.64.1, 00000000004.53.1] 2020-11-19T03:17:33.334 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:33.323+0000 7fd8587b80c0 0 bucket checkpoint complete 2020-11-19T03:17:33.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:33.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:33.366 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:33.415 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:33.415 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:33.664 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:33.657+0000 7f0e7e9410c0 1 bucket sync caught up with source: 2020-11-19T03:17:33.665 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000006.53.1, 00000000004.48.1, 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, 00000000004.40.1, 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, 00000000004.64.1, 00000000004.53.1] 2020-11-19T03:17:33.665 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000006.53.1, 00000000004.48.1, 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, 00000000004.40.1, 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, 00000000004.64.1, 00000000004.53.1] 2020-11-19T03:17:33.665 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:33.657+0000 7f0e7e9410c0 0 bucket checkpoint complete 2020-11-19T03:17:33.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:33.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:33.695 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:33.762 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:33.763 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:34.017 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:34.007+0000 7f4408d080c0 1 bucket sync caught up with source: 2020-11-19T03:17:34.017 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.53.1, 00000000004.48.1, 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, 00000000004.40.1, 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, 00000000004.64.1, 00000000004.53.1] 2020-11-19T03:17:34.017 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.53.1, 00000000004.48.1, 00000000006.91.1, 00000000004.56.1, 00000000006.71.1, 00000000004.40.1, 00000000004.49.1, 00000000006.63.1, 00000000004.65.1, 00000000004.64.1, 00000000004.53.1] 2020-11-19T03:17:34.018 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:34.007+0000 7f4408d080c0 0 bucket checkpoint complete 2020-11-19T03:17:34.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:34.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:34.127 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:34.182 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:34.183 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:34.450 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:34.445+0000 7ffa0e6750c0 1 waiting for incremental sync to catch up: 2020-11-19T03:17:34.451 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000003.54.1, , 00000000003.92.1, , 00000000003.56.1, , , 00000000003.62.1, , , ] 2020-11-19T03:17:34.451 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.58.5, 00000000002.52.1, 00000000004.96.5, 00000000002.66.1, 00000000004.60.5, 00000000002.43.1, 00000000002.40.1, 00000000004.69.5, 00000000002.72.1, 00000000002.65.1, 00000000002.58.1] 2020-11-19T03:17:39.455 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:39.449+0000 7ffa0e6750c0 1 waiting for incremental sync to catch up: 2020-11-19T03:17:39.455 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000003.54.1, , 00000000003.92.1, , 00000000003.56.1, , , 00000000003.62.1, , , ] 2020-11-19T03:17:39.456 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.58.5, 00000000002.52.1, 00000000004.96.5, 00000000002.66.1, 00000000004.60.5, 00000000002.43.1, 00000000002.40.1, 00000000004.69.5, 00000000002.72.1, 00000000002.65.1, 00000000002.58.1] 2020-11-19T03:17:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:44.453+0000 7ffa0e6750c0 1 bucket sync caught up with source: 2020-11-19T03:17:44.786 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.58.5, 00000000002.52.1, 00000000004.96.5, 00000000002.66.1, 00000000004.60.5, 00000000002.43.1, 00000000002.40.1, 00000000004.69.5, 00000000002.72.1, 00000000002.65.1, 00000000002.58.1] 2020-11-19T03:17:44.787 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.58.5, 00000000002.52.1, 00000000004.96.5, 00000000002.66.1, 00000000004.60.5, 00000000002.43.1, 00000000002.40.1, 00000000004.69.5, 00000000002.72.1, 00000000002.65.1, 00000000002.58.1] 2020-11-19T03:17:44.787 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:44.453+0000 7ffa0e6750c0 0 bucket checkpoint complete 2020-11-19T03:17:44.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:44.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:44.789 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:44.836 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:44.836 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:45.087 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:45.081+0000 7f96a04af0c0 1 bucket sync caught up with source: 2020-11-19T03:17:45.087 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.58.5, 00000000002.52.1, 00000000004.96.5, 00000000002.66.1, 00000000004.60.5, 00000000002.43.1, 00000000002.40.1, 00000000004.69.5, 00000000002.72.1, 00000000002.65.1, 00000000002.58.1] 2020-11-19T03:17:45.088 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.58.5, 00000000002.52.1, 00000000004.96.5, 00000000002.66.1, 00000000004.60.5, 00000000002.43.1, 00000000002.40.1, 00000000004.69.5, 00000000002.72.1, 00000000002.65.1, 00000000002.58.1] 2020-11-19T03:17:45.088 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:45.081+0000 7f96a04af0c0 0 bucket checkpoint complete 2020-11-19T03:17:45.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:45.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:45.120 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:45.205 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:45.206 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:45.474 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:45.469+0000 7f33fec6a0c0 1 bucket sync caught up with source: 2020-11-19T03:17:45.474 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.56.1, 00000000002.51.1, 00000000004.95.1, 00000000002.62.1, 00000000004.58.1, 00000000002.43.1, 00000000002.42.1, 00000000004.66.1, 00000000002.74.1, 00000000002.69.1, 00000000002.54.1] 2020-11-19T03:17:45.474 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.56.1, 00000000002.51.1, 00000000004.95.1, 00000000002.62.1, 00000000004.58.1, 00000000002.43.1, 00000000002.42.1, 00000000004.66.1, 00000000002.74.1, 00000000002.69.1, 00000000002.54.1] 2020-11-19T03:17:45.475 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:45.469+0000 7f33fec6a0c0 0 bucket checkpoint complete 2020-11-19T03:17:45.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:45.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:45.503 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:45.553 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:45.554 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:45.822 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:45.815+0000 7fe19a6ad0c0 1 bucket sync caught up with source: 2020-11-19T03:17:45.823 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.56.1, 00000000002.51.1, 00000000004.95.1, 00000000002.62.1, 00000000004.58.1, 00000000002.43.1, 00000000002.42.1, 00000000004.66.1, 00000000002.74.1, 00000000002.69.1, 00000000002.54.1] 2020-11-19T03:17:45.823 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.56.1, 00000000002.51.1, 00000000004.95.1, 00000000002.62.1, 00000000004.58.1, 00000000002.43.1, 00000000002.42.1, 00000000004.66.1, 00000000002.74.1, 00000000002.69.1, 00000000002.54.1] 2020-11-19T03:17:45.824 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:45.815+0000 7fe19a6ad0c0 0 bucket checkpoint complete 2020-11-19T03:17:45.854 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-62 zones={test-zone2, test-zone1} 2020-11-19T03:17:45.866 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:17:45.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:45.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:45.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:45.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:45.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:17:45.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:17:45.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:17:45.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:17:45.876 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:17:45.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:45.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:45.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:45.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:45.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:17:45.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:17:45.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:17:45.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:17:45.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:45.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:46.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:46.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:46.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:46.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:46.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:46.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:46.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:46.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:46.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:46.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:47.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:47.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:47.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:47.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:47.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:47.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:47.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:47.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:47.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:48.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:48.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:48.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:48.416 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-62 zones={test-zone2, test-zone1} 2020-11-19T03:17:48.416 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-62 zones={test-zone3, test-zone1} 2020-11-19T03:17:48.427 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:17:48.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:48.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:48.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:48.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:48.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:17:48.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:17:48.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:17:48.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:17:48.436 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:17:48.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:48.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:48.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:48.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:48.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:17:48.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:17:48.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:17:48.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:17:48.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:48.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:48.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:48.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:49.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:49.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:49.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:49.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:50.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:50.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:50.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:50.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:50.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:50.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:50.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:50.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:50.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:50.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:50.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:51.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:51.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:51.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:51.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:51.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:51.638 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-62 zones={test-zone3, test-zone1} 2020-11-19T03:17:51.639 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-62 zones={test-zone3, test-zone2} 2020-11-19T03:17:51.651 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:17:51.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:51.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:51.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:51.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:51.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:17:51.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:17:51.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:17:51.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:17:51.660 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:17:51.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:51.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:51.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:51.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:51.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:17:51.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:17:51.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:17:51.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:17:51.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:51.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:51.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:51.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:52.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:52.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:52.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:52.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:52.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:52.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:52.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:52.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:52.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:53.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:53.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:53.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:53.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:53.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:53.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:53.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:54.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:54.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:54.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:54.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:54.500 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-62 zones={test-zone3, test-zone2} 2020-11-19T03:17:54.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:54.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:54.505 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:54.556 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:54.557 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:54.824 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:54.811+0000 7f3cbffb60c0 1 bucket sync caught up with source: 2020-11-19T03:17:54.824 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.52.5, 00000000004.66.1, 00000000006.100.5, 00000000004.59.1, 00000000006.53.5, 00000000004.33.1, 00000000004.44.1, 00000000006.57.5, 00000000004.64.1, 00000000004.53.1, 00000000004.69.1] 2020-11-19T03:17:54.825 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.52.5, 00000000004.66.1, 00000000006.100.5, 00000000004.59.1, 00000000006.53.5, 00000000004.33.1, 00000000004.44.1, 00000000006.57.5, 00000000004.64.1, 00000000004.53.1, 00000000004.69.1] 2020-11-19T03:17:54.825 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:54.811+0000 7f3cbffb60c0 0 bucket checkpoint complete 2020-11-19T03:17:54.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:54.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:54.855 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:54.907 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:55.141 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:55.133+0000 7f7b6ddcf0c0 1 bucket sync caught up with source: 2020-11-19T03:17:55.141 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000006.52.5, 00000000004.66.1, 00000000006.100.5, 00000000004.59.1, 00000000006.53.5, 00000000004.33.1, 00000000004.44.1, 00000000006.57.5, 00000000004.64.1, 00000000004.53.1, 00000000004.69.1] 2020-11-19T03:17:55.142 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000006.52.5, 00000000004.66.1, 00000000006.100.5, 00000000004.59.1, 00000000006.53.5, 00000000004.33.1, 00000000004.44.1, 00000000006.57.5, 00000000004.64.1, 00000000004.53.1, 00000000004.69.1] 2020-11-19T03:17:55.142 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:55.133+0000 7f7b6ddcf0c0 0 bucket checkpoint complete 2020-11-19T03:17:55.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:55.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:55.171 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:55.220 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:55.220 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:55.465 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:55.455+0000 7f7fbb0c20c0 1 bucket sync caught up with source: 2020-11-19T03:17:55.466 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.52.5, 00000000004.66.1, 00000000006.100.5, 00000000004.59.1, 00000000006.53.5, 00000000004.33.1, 00000000004.44.1, 00000000006.57.5, 00000000004.64.1, 00000000004.53.1, 00000000004.69.1] 2020-11-19T03:17:55.466 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.52.5, 00000000004.66.1, 00000000006.100.5, 00000000004.59.1, 00000000006.53.5, 00000000004.33.1, 00000000004.44.1, 00000000006.57.5, 00000000004.64.1, 00000000004.53.1, 00000000004.69.1] 2020-11-19T03:17:55.466 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:55.455+0000 7f7fbb0c20c0 0 bucket checkpoint complete 2020-11-19T03:17:55.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:55.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:55.590 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:55.640 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:55.640 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:55.901+0000 7fdd912290c0 1 bucket sync caught up with source: 2020-11-19T03:17:55.908 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.51.1, 00000000002.61.1, 00000000004.102.1, 00000000002.52.1, 00000000004.42.1, 00000000002.31.1, 00000000002.40.1, 00000000004.53.1, 00000000002.83.1, 00000000002.55.1, 00000000002.79.1] 2020-11-19T03:17:55.908 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.51.1, 00000000002.61.1, 00000000004.102.1, 00000000002.52.1, 00000000004.42.1, 00000000002.31.1, 00000000002.40.1, 00000000004.53.1, 00000000002.83.1, 00000000002.55.1, 00000000002.79.1] 2020-11-19T03:17:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:55.901+0000 7fdd912290c0 0 bucket checkpoint complete 2020-11-19T03:17:55.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:55.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:55.941 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:55.986 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:55.987 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:56.242 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:56.237+0000 7fd21fd800c0 1 bucket sync caught up with source: 2020-11-19T03:17:56.242 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.51.1, 00000000002.61.1, 00000000004.102.1, 00000000002.52.1, 00000000004.42.1, 00000000002.31.1, 00000000002.40.1, 00000000004.53.1, 00000000002.83.1, 00000000002.55.1, 00000000002.79.1] 2020-11-19T03:17:56.242 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.51.1, 00000000002.61.1, 00000000004.102.1, 00000000002.52.1, 00000000004.42.1, 00000000002.31.1, 00000000002.40.1, 00000000004.53.1, 00000000002.83.1, 00000000002.55.1, 00000000002.79.1] 2020-11-19T03:17:56.243 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:56.237+0000 7fd21fd800c0 0 bucket checkpoint complete 2020-11-19T03:17:56.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:56.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:56.273 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:56.321 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:56.321 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:56.580 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:56.573+0000 7f3b956df0c0 1 bucket sync caught up with source: 2020-11-19T03:17:56.581 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.50.1, 00000000002.65.1, 00000000004.113.1, 00000000002.60.1, 00000000004.48.1, 00000000002.33.1, 00000000002.36.1, 00000000004.55.1, 00000000002.63.1, 00000000002.58.1, 00000000002.84.1] 2020-11-19T03:17:56.581 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.50.1, 00000000002.65.1, 00000000004.113.1, 00000000002.60.1, 00000000004.48.1, 00000000002.33.1, 00000000002.36.1, 00000000004.55.1, 00000000002.63.1, 00000000002.58.1, 00000000002.84.1] 2020-11-19T03:17:56.581 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:17:56.573+0000 7f3b956df0c0 0 bucket checkpoint complete 2020-11-19T03:17:56.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:56.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:17:56.611 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:17:56.658 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:17:56.659 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:17:56.934 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:56.923+0000 7f7c517230c0 1 bucket sync caught up with source: 2020-11-19T03:17:56.934 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.50.1, 00000000002.65.1, 00000000004.113.1, 00000000002.60.1, 00000000004.48.1, 00000000002.33.1, 00000000002.36.1, 00000000004.55.1, 00000000002.63.1, 00000000002.58.1, 00000000002.84.1] 2020-11-19T03:17:56.934 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.50.1, 00000000002.65.1, 00000000004.113.1, 00000000002.60.1, 00000000004.48.1, 00000000002.33.1, 00000000002.36.1, 00000000004.55.1, 00000000002.63.1, 00000000002.58.1, 00000000002.84.1] 2020-11-19T03:17:56.934 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:17:56.923+0000 7f7c517230c0 0 bucket checkpoint complete 2020-11-19T03:17:56.965 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-63 zones={test-zone2, test-zone1} 2020-11-19T03:17:56.993 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:17:57.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:57.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:57.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:57.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:57.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:17:57.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:17:57.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:17:57.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:17:57.013 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:17:57.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:17:57.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:17:57.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:17:57.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:17:57.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:17:57.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:17:57.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:17:57.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:17:57.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:57.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:57.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:17:57.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:57.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:57.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:17:57.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:57.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:57.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:17:58.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:58.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:58.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:17:58.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:58.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:58.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:17:58.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:59.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:59.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:17:59.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:59.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:59.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:17:59.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:17:59.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:00.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:00.166 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-63 zones={test-zone2, test-zone1} 2020-11-19T03:18:00.167 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-63 zones={test-zone3, test-zone1} 2020-11-19T03:18:00.175 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:18:00.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:00.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:00.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:00.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:00.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:00.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:00.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:00.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:00.183 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:18:00.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:00.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:00.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:00.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:00.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:00.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:00.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:00.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:00.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:00.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:00.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:00.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:00.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:00.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:00.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:00.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:00.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:01.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:01.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:01.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:01.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:01.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:01.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:01.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:01.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:01.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:01.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:01.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:02.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:02.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:02.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:02.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:02.475 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-63 zones={test-zone3, test-zone1} 2020-11-19T03:18:02.476 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-63 zones={test-zone3, test-zone2} 2020-11-19T03:18:02.492 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:18:02.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:02.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:02.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:02.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:02.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:02.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:02.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:02.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:02.502 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:18:02.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:02.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:02.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:02.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:02.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:02.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:02.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:02.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:02.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:02.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:02.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:02.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:02.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:03.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:03.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:03.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:03.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:03.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:03.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:03.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:03.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:04.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:04.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:04.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:04.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:04.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:05.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:05.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:05.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:05.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:05.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:05.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:05.594 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-63 zones={test-zone3, test-zone2} 2020-11-19T03:18:05.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:05.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:05.596 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:05.643 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:05.644 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:05.904 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:05.895+0000 7fae7c5260c0 1 bucket sync caught up with source: 2020-11-19T03:18:05.905 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.57.5, 00000000004.51.1, 00000000006.95.5, 00000000004.58.1, 00000000006.41.5, 00000000004.68.1, 00000000004.67.1, 00000000006.52.5, 00000000004.86.1, 00000000004.55.1, 00000000004.76.1] 2020-11-19T03:18:05.905 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.57.5, 00000000004.51.1, 00000000006.95.5, 00000000004.58.1, 00000000006.41.5, 00000000004.68.1, 00000000004.67.1, 00000000006.52.5, 00000000004.86.1, 00000000004.55.1, 00000000004.76.1] 2020-11-19T03:18:05.905 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:05.895+0000 7fae7c5260c0 0 bucket checkpoint complete 2020-11-19T03:18:05.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:05.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:05.938 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:05.985 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:05.986 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:06.231 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:06.224+0000 7f93974930c0 1 bucket sync caught up with source: 2020-11-19T03:18:06.231 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000006.57.5, 00000000004.51.1, 00000000006.95.5, 00000000004.58.1, 00000000006.41.5, 00000000004.68.1, 00000000004.67.1, 00000000006.52.5, 00000000004.86.1, 00000000004.55.1, 00000000004.76.1] 2020-11-19T03:18:06.232 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000006.57.5, 00000000004.51.1, 00000000006.95.5, 00000000004.58.1, 00000000006.41.5, 00000000004.68.1, 00000000004.67.1, 00000000006.52.5, 00000000004.86.1, 00000000004.55.1, 00000000004.76.1] 2020-11-19T03:18:06.232 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:06.224+0000 7f93974930c0 0 bucket checkpoint complete 2020-11-19T03:18:06.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:06.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:06.260 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:06.312 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:06.313 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:06.546 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:06.535+0000 7f94742540c0 1 bucket sync caught up with source: 2020-11-19T03:18:06.546 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000006.57.5, 00000000004.51.1, 00000000006.95.5, 00000000004.58.1, 00000000006.41.5, 00000000004.68.1, 00000000004.67.1, 00000000006.52.5, 00000000004.86.1, 00000000004.55.1, 00000000004.76.1] 2020-11-19T03:18:06.546 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000006.57.5, 00000000004.51.1, 00000000006.95.5, 00000000004.58.1, 00000000006.41.5, 00000000004.68.1, 00000000004.67.1, 00000000006.52.5, 00000000004.86.1, 00000000004.55.1, 00000000004.76.1] 2020-11-19T03:18:06.547 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:06.535+0000 7f94742540c0 0 bucket checkpoint complete 2020-11-19T03:18:06.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:06.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:06.675 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:06.723 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:06.723 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:06.968+0000 7ff24f04c0c0 1 bucket sync caught up with source: 2020-11-19T03:18:06.977 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.62.5, 00000000002.41.1, 00000000004.98.5, 00000000002.64.1, 00000000004.37.5, 00000000002.58.1, 00000000002.70.1, 00000000004.48.5, 00000000002.75.1, 00000000002.53.1, 00000000002.76.1] 2020-11-19T03:18:06.977 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.62.5, 00000000002.41.1, 00000000004.98.5, 00000000002.64.1, 00000000004.37.5, 00000000002.58.1, 00000000002.70.1, 00000000004.48.5, 00000000002.75.1, 00000000002.53.1, 00000000002.76.1] 2020-11-19T03:18:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:06.968+0000 7ff24f04c0c0 0 bucket checkpoint complete 2020-11-19T03:18:07.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:07.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:07.011 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:07.060 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:07.060 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:07.318 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:07.312+0000 7f2a535040c0 1 bucket sync caught up with source: 2020-11-19T03:18:07.318 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.62.5, 00000000002.41.1, 00000000004.98.5, 00000000002.64.1, 00000000004.37.5, 00000000002.58.1, 00000000002.70.1, 00000000004.48.5, 00000000002.75.1, 00000000002.53.1, 00000000002.76.1] 2020-11-19T03:18:07.318 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.62.5, 00000000002.41.1, 00000000004.98.5, 00000000002.64.1, 00000000004.37.5, 00000000002.58.1, 00000000002.70.1, 00000000004.48.5, 00000000002.75.1, 00000000002.53.1, 00000000002.76.1] 2020-11-19T03:18:07.318 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:07.312+0000 7f2a535040c0 0 bucket checkpoint complete 2020-11-19T03:18:07.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:07.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:07.351 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:07.700+0000 7fed717f40c0 1 bucket sync caught up with source: 2020-11-19T03:18:07.710 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [00000000004.53.1, 00000000002.47.1, 00000000004.94.1, 00000000002.60.1, 00000000004.36.1, 00000000002.64.1, 00000000002.72.1, 00000000004.52.1, 00000000002.71.1, 00000000002.58.1, 00000000002.103.1] 2020-11-19T03:18:07.710 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [00000000004.53.1, 00000000002.47.1, 00000000004.94.1, 00000000002.60.1, 00000000004.36.1, 00000000002.64.1, 00000000002.72.1, 00000000004.52.1, 00000000002.71.1, 00000000002.58.1, 00000000002.103.1] 2020-11-19T03:18:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:07.704+0000 7fed717f40c0 0 bucket checkpoint complete 2020-11-19T03:18:07.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:07.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:07.741 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:07.793 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:07.793 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:08.051+0000 7f0764e420c0 1 bucket sync caught up with source: 2020-11-19T03:18:08.058 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000004.53.1, 00000000002.47.1, 00000000004.94.1, 00000000002.60.1, 00000000004.36.1, 00000000002.64.1, 00000000002.72.1, 00000000004.52.1, 00000000002.71.1, 00000000002.58.1, 00000000002.103.1] 2020-11-19T03:18:08.059 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000004.53.1, 00000000002.47.1, 00000000004.94.1, 00000000002.60.1, 00000000004.36.1, 00000000002.64.1, 00000000002.72.1, 00000000004.52.1, 00000000002.71.1, 00000000002.58.1, 00000000002.103.1] 2020-11-19T03:18:08.059 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:08.051+0000 7f0764e420c0 0 bucket checkpoint complete 2020-11-19T03:18:08.090 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-64 zones={test-zone2, test-zone1} 2020-11-19T03:18:08.101 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:18:08.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:08.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:08.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:08.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:08.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:08.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:08.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:08.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:08.111 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:18:08.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:08.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:08.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:08.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:08.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:08.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:08.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:08.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:08.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:08.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:08.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:08.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:08.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:08.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:09.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:09.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:09.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:10.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:10.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:10.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:10.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:10.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:10.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:10.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:10.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:10.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:10.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:10.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:11.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:11.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:11.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:11.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:11.565 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-64 zones={test-zone2, test-zone1} 2020-11-19T03:18:11.565 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-64 zones={test-zone3, test-zone1} 2020-11-19T03:18:11.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:18:11.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:11.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:11.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:11.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:11.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:11.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:11.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:11.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:11.597 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:18:11.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:11.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:11.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:11.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:11.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:11.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:11.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:11.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:11.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:11.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:11.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:11.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:11.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:12.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:12.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:12.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:12.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:12.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:12.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:12.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:12.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:12.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:13.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:13.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:13.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:13.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:13.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:13.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:14.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:14.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:14.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:14.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:14.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-64 zones={test-zone3, test-zone1} 2020-11-19T03:18:14.763 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-64 zones={test-zone3, test-zone2} 2020-11-19T03:18:14.776 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:18:14.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:14.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:14.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:14.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:14.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:14.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:14.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:14.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:14.788 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:18:14.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-19T03:18:14.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-19T03:18:14.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-19T03:18:14.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-19T03:18:14.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-19T03:18:14.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-19T03:18:14.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-19T03:18:14.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-19T03:18:14.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:14.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:15.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-19T03:18:15.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:15.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:15.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-19T03:18:15.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:15.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:15.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-19T03:18:15.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:15.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:15.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-19T03:18:15.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:16.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:16.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-19T03:18:16.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:16.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:16.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-19T03:18:16.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:16.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:16.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-19T03:18:16.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:16.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:17.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-19T03:18:17.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-64 zones={test-zone3, test-zone2} 2020-11-19T03:18:17.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:17.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:17.211 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:17.539 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:18:17.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:17.540 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-19T03:18:17.540 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:18.016089Z" 2020-11-19T03:18:17.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:17.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:17.540 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-19T03:18:17.541 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:37.079841Z" 2020-11-19T03:18:17.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:17.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:17.541 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-19T03:18:17.542 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:37.032841Z" 2020-11-19T03:18:17.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:17.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:17.542 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000018978", 2020-11-19T03:18:17.542 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:36.992088Z" 2020-11-19T03:18:17.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:17.543 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:18:17.544 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:18:17.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:17.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:17.545 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:17.589 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:17.590 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:17.872 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:18:17.873 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:18:17.873 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:18:17.874 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:18:17.874 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:18:17.875 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16121403893923359777 2020-11-19T03:18:17.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:17.875 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:18:17.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:17.877 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:18:17.877 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:17.878 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:17.879 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-19T03:18:17.880 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:17.881 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:17.881 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:17.881 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:18.016089Z" 2020-11-19T03:18:17.881 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:17.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:17.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:17.882 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:18:17.882 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:17.882 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:17.883 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-19T03:18:17.883 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:17.883 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:17.883 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:17.883 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:37.079841Z" 2020-11-19T03:18:17.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:17.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:17.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:17.884 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:18:17.884 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:17.885 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:17.885 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-19T03:18:17.885 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:17.885 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:17.885 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:17.886 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:37.032841Z" 2020-11-19T03:18:17.886 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:17.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:17.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:17.886 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:18:17.887 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:17.887 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:17.887 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000018978", 2020-11-19T03:18:17.887 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:17.887 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:17.888 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:17.888 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:36.992088Z" 2020-11-19T03:18:17.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:17.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:17.888 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:18:17.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:17.889 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:18:17.889 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:18:17.889 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:18:17.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:17.890 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:18:17.891 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:18:17.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:17.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:17.892 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:17.940 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:17.940 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:18.234 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:18:18.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:18.235 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-19T03:18:18.235 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:18.016089Z" 2020-11-19T03:18:18.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:18.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:18.236 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-19T03:18:18.236 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:37.079841Z" 2020-11-19T03:18:18.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:18.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:18.237 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-19T03:18:18.237 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:37.032841Z" 2020-11-19T03:18:18.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:18.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:18.238 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000018978", 2020-11-19T03:18:18.239 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:36.992088Z" 2020-11-19T03:18:18.239 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:18.239 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:18:18.241 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:18:18.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:18.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:18.243 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:18.329 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:18.329 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:18.590 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:18:18.591 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:18:18.591 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:18:18.591 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:18:18.591 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:18:18.592 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7000157538207389965 2020-11-19T03:18:18.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:18.592 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:18:18.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:18.593 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:18:18.593 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:18.593 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:18.596 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-19T03:18:18.596 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:18.596 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:18.596 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:18.597 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:18.016089Z" 2020-11-19T03:18:18.597 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:18.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:18.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:18.598 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:18:18.598 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:18.598 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:18.598 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-19T03:18:18.598 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:18.599 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:18.599 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:18.599 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:37.079841Z" 2020-11-19T03:18:18.599 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:18.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:18.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:18.600 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:18:18.600 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:18.600 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:18.601 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-19T03:18:18.601 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:18.601 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:18.601 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:18.601 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:37.032841Z" 2020-11-19T03:18:18.602 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:18.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:18.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:18.602 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:18:18.602 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:18.603 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:18.603 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000018978", 2020-11-19T03:18:18.603 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:18.603 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:18.603 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:18.604 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:36.992088Z" 2020-11-19T03:18:18.604 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:18.604 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:18.604 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:18:18.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:18.605 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:18:18.605 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:18:18.605 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:18:18.605 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:18.606 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:18:18.607 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:18:18.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:18.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:18.608 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:18.705 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:19.345 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:19.347 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:18:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-19T03:18:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:18.016089Z" 2020-11-19T03:18:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-19T03:18:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:37.079841Z" 2020-11-19T03:18:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-19T03:18:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:37.032841Z" 2020-11-19T03:18:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000018978", 2020-11-19T03:18:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:36.992088Z" 2020-11-19T03:18:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:19.350 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:18:19.351 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:18:19.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:19.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:19.352 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:19.396 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:19.396 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:19.666 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:18:19.667 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:18:19.667 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:18:19.667 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:18:19.668 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:18:19.668 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 12326036592617550149 2020-11-19T03:18:19.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:19.668 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:18:19.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:19.669 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:18:19.669 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:19.669 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:19.669 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-19T03:18:19.670 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:19.670 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:19.670 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:19.670 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:18.016089Z" 2020-11-19T03:18:19.670 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:19.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:19.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:19.671 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:18:19.671 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:19.671 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:19.672 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-19T03:18:19.672 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:19.672 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:19.672 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:19.672 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:37.079841Z" 2020-11-19T03:18:19.673 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:19.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:19.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:19.673 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:18:19.674 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:19.674 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:19.674 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-19T03:18:19.674 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:19.674 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:19.675 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:19.675 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:37.032841Z" 2020-11-19T03:18:19.675 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:19.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:19.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:19.676 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:18:19.676 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:19.676 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:19.676 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000018978", 2020-11-19T03:18:19.676 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:19.677 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:19.677 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:19.677 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:36.992088Z" 2020-11-19T03:18:19.677 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:19.677 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:19.678 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:18:19.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:19.678 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:18:19.678 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:18:19.678 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:18:19.679 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:19.679 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:18:19.680 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:18:19.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:19.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:19.681 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:19.768 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:19.769 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:20.038 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:18:20.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:20.039 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000009278", 2020-11-19T03:18:20.040 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:17:39.232363Z" 2020-11-19T03:18:20.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:20.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:20.041 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005701", 2020-11-19T03:18:20.041 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:17:39.236053Z" 2020-11-19T03:18:20.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:20.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:20.043 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006092", 2020-11-19T03:18:20.043 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:17:39.241665Z" 2020-11-19T03:18:20.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:20.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:20.045 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000010209", 2020-11-19T03:18:20.046 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:17:39.245294Z" 2020-11-19T03:18:20.046 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:20.046 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:18:20.047 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:18:20.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:20.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:20.049 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:20.093 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:18:20.380 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:18:20.380 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:18:20.381 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:18:20.381 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:18:20.381 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 7306459910885960034 2020-11-19T03:18:20.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:20.381 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:18:20.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:20.382 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:18:20.382 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:20.382 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:20.383 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000009278", 2020-11-19T03:18:20.383 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:20.383 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:20.383 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:20.384 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:39.232363Z" 2020-11-19T03:18:20.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:20.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:20.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:18:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005701", 2020-11-19T03:18:20.386 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:20.386 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:20.386 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:20.387 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:39.236053Z" 2020-11-19T03:18:20.387 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:20.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:20.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:18:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006092", 2020-11-19T03:18:20.388 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:20.389 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:20.390 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:20.390 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:39.241665Z" 2020-11-19T03:18:20.390 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:20.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:20.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:20.391 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:18:20.391 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:20.392 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:20.392 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000010209", 2020-11-19T03:18:20.392 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:20.393 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:20.393 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:20.393 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:39.245294Z" 2020-11-19T03:18:20.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:20.394 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:20.394 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:18:20.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:20.394 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:18:20.395 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:18:20.395 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:18:20.395 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:20.395 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:18:20.397 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:18:20.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:20.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:20.398 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:20.438 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:20.438 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:20.731 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2020-11-19T03:18:20.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:20.731 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000009278", 2020-11-19T03:18:20.731 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:17:39.232363Z" 2020-11-19T03:18:20.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:20.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:20.732 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005701", 2020-11-19T03:18:20.732 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:17:39.236053Z" 2020-11-19T03:18:20.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:20.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:20.733 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006092", 2020-11-19T03:18:20.733 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:17:39.241665Z" 2020-11-19T03:18:20.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:20.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:20.734 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000010209", 2020-11-19T03:18:20.734 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2020-11-19T03:17:39.245294Z" 2020-11-19T03:18:20.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:20.734 INFO:teuthology.orchestra.run.smithi160.stdout:] 2020-11-19T03:18:20.735 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:18:20.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:20.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:20.737 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:20.778 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:20.778 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:21.055 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:18:21.055 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:18:21.056 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:18:21.056 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:18:21.056 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:18:21.056 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 597164135494128594 2020-11-19T03:18:21.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.057 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:18:21.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.057 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:18:21.057 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:21.058 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:21.058 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000009278", 2020-11-19T03:18:21.058 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:21.058 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:21.059 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:21.059 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:39.232363Z" 2020-11-19T03:18:21.059 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.060 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:18:21.060 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:21.060 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:21.060 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000005701", 2020-11-19T03:18:21.060 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:21.061 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:21.061 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:21.061 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:39.236053Z" 2020-11-19T03:18:21.061 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.062 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:18:21.062 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:21.062 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:21.062 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006092", 2020-11-19T03:18:21.063 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:21.063 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:21.063 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:21.063 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:39.241665Z" 2020-11-19T03:18:21.063 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:18:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000010209", 2020-11-19T03:18:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:39.245294Z" 2020-11-19T03:18:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:18:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:18:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:18:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:18:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.067 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:18:21.069 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:18:21.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:21.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:21.070 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:21.147 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:21.147 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:21.427 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:18:21.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.428 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000009944", 2020-11-19T03:18:21.428 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:34.976937Z" 2020-11-19T03:18:21.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.429 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006368", 2020-11-19T03:18:21.429 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:34.981986Z" 2020-11-19T03:18:21.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.430 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006491", 2020-11-19T03:18:21.430 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:34.986942Z" 2020-11-19T03:18:21.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.432 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000010608", 2020-11-19T03:18:21.432 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:34.991471Z" 2020-11-19T03:18:21.432 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.433 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:18:21.434 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:18:21.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:21.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:21.436 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:21.794 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:18:21.795 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:18:21.795 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:18:21.795 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:18:21.796 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:18:21.796 INFO:teuthology.orchestra.run.smithi086.stdout: "instance_id": 2448524765768921960 2020-11-19T03:18:21.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.796 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:18:21.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.797 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:18:21.797 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:21.797 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:21.798 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000009944", 2020-11-19T03:18:21.798 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:21.798 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:21.798 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:21.798 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:34.976937Z" 2020-11-19T03:18:21.799 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.799 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:18:21.800 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:21.800 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:21.800 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006368", 2020-11-19T03:18:21.800 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:21.800 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:21.801 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:21.801 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:34.981986Z" 2020-11-19T03:18:21.801 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.802 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:18:21.802 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:21.802 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:21.802 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006491", 2020-11-19T03:18:21.802 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:21.803 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:21.803 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:21.803 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:34.986942Z" 2020-11-19T03:18:21.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:21.804 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:18:21.804 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:18:21.804 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "incremental-sync", 2020-11-19T03:18:21.804 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000010608", 2020-11-19T03:18:21.805 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:18:21.805 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:18:21.805 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:18:21.805 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:17:34.991471Z" 2020-11-19T03:18:21.805 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.806 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.806 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:18:21.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:21.806 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:18:21.806 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 0, 2020-11-19T03:18:21.807 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 0 2020-11-19T03:18:21.807 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:21.807 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:18:21.808 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:18:21.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:21.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:21.810 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:21.891 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:21.891 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:22.145 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:18:22.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:22.145 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000009944", 2020-11-19T03:18:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:34.976937Z" 2020-11-19T03:18:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006368", 2020-11-19T03:18:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:34.981986Z" 2020-11-19T03:18:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000006491", 2020-11-19T03:18:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:34.986942Z" 2020-11-19T03:18:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:18:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:18:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "00000000000000000000:00000000000000010608", 2020-11-19T03:18:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:17:34.991471Z" 2020-11-19T03:18:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:18:22.149 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:18:22.150 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:18:22.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:22.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:18:22.152 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:18:22.203 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:22.204 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:22.465 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:18:22.465 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:18:22.466 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:18:22.466 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:18:22.466 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:18:22.466 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 16483361435957904979 2020-11-19T03:18:22.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:22.467 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:18:22.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:22.467 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:18:22.467 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:22.468 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:22.468 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000009944", 2020-11-19T03:18:22.468 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:22.468 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:22.468 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:22.469 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:34.976937Z" 2020-11-19T03:18:22.469 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:22.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:22.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:22.470 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:18:22.470 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:22.470 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:22.470 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006368", 2020-11-19T03:18:22.471 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:22.471 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:22.471 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:22.471 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:34.981986Z" 2020-11-19T03:18:22.472 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:22.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:22.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:22.472 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:18:22.472 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:22.473 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:22.473 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006491", 2020-11-19T03:18:22.473 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:22.473 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:22.473 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:22.474 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:34.986942Z" 2020-11-19T03:18:22.474 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:22.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:22.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:18:22.475 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:18:22.475 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:18:22.475 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2020-11-19T03:18:22.475 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000010608", 2020-11-19T03:18:22.475 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:18:22.476 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:18:22.476 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:18:22.476 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:17:34.991471Z" 2020-11-19T03:18:22.476 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:22.476 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:22.477 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:18:22.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:18:22.477 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:18:22.477 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2020-11-19T03:18:22.477 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2020-11-19T03:18:22.478 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:18:22.478 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:18:22.479 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:18:22.479 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2020-11-19T03:18:22.487 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-65 2020-11-19T03:18:22.533 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=atsuux-66 2020-11-19T03:18:22.604 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=atsuux-67 2020-11-19T03:18:25.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:25.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:25.148 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:25.196 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:25.196 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:25.441 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:25.435+0000 7f31ca78a0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:18:25.441 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.99.5, , , 00000000001.47.5, , , , , ] 2020-11-19T03:18:25.442 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:30.445 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:30.439+0000 7f31ca78a0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:18:30.445 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.99.5, , , 00000000001.47.5, , , , , ] 2020-11-19T03:18:30.445 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:35.449 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:35.443+0000 7f31ca78a0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:18:35.449 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.99.5, , , 00000000001.47.5, , , , , ] 2020-11-19T03:18:35.450 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:40.453 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:40.446+0000 7f31ca78a0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:18:40.453 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.99.5, , , 00000000001.47.5, , , , , ] 2020-11-19T03:18:40.454 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:45.458 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:45.450+0000 7f31ca78a0c0 1 bucket sync caught up with source: 2020-11-19T03:18:45.464 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:45.464 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:45.465 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:45.450+0000 7f31ca78a0c0 0 bucket checkpoint complete 2020-11-19T03:18:45.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:45.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:45.494 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:45.542 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:45.542 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:45.819+0000 7f288b87b0c0 1 bucket sync caught up with source: 2020-11-19T03:18:45.824 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:45.824 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:45.819+0000 7f288b87b0c0 0 bucket checkpoint complete 2020-11-19T03:18:45.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:45.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:45.852 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:45.900 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:45.900 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:46.140 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:46.134+0000 7fbd61bc40c0 1 bucket sync caught up with source: 2020-11-19T03:18:46.140 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:46.140 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000001.99.5, , , 00000000010.57.16, , , , , ] 2020-11-19T03:18:46.141 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:46.134+0000 7fbd61bc40c0 0 bucket checkpoint complete 2020-11-19T03:18:46.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:46.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:46.274 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:46.322 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:46.322 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:46.587+0000 7f58e60160c0 1 bucket sync caught up with source: 2020-11-19T03:18:46.592 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000001.65.5, , , , , ] 2020-11-19T03:18:46.592 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000001.65.5, , , , , ] 2020-11-19T03:18:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:46.587+0000 7f58e60160c0 0 bucket checkpoint complete 2020-11-19T03:18:46.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:46.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:46.624 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:46.938 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:46.935+0000 7f61236e00c0 1 bucket sync caught up with source: 2020-11-19T03:18:46.939 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000001.65.5, , , , , ] 2020-11-19T03:18:46.939 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000001.65.5, , , , , ] 2020-11-19T03:18:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:46.935+0000 7f61236e00c0 0 bucket checkpoint complete 2020-11-19T03:18:46.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:46.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:46.971 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:47.018 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:47.019 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:47.279+0000 7fe9dfd2d0c0 1 bucket sync caught up with source: 2020-11-19T03:18:47.282 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:18:47.282 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:18:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:18:47.279+0000 7fe9dfd2d0c0 0 bucket checkpoint complete 2020-11-19T03:18:47.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:47.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:18:47.312 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:18:47.359 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:18:47.359 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:18:47.626 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:47.622+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:18:47.626 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:18:47.627 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:18:52.630 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:52.626+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:18:52.631 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:18:52.631 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:18:57.634 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:18:57.630+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:18:57.635 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:18:57.635 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:02.639 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:02.634+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:19:02.639 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:19:02.640 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:07.642 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:07.634+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:19:07.642 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:19:07.643 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:12.646 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:12.642+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:19:12.646 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:19:12.647 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:17.650 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:17.642+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:19:17.651 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:19:17.651 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:22.650 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:22.642+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:19:22.802 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:19:22.802 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:27.653 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:27.646+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:19:28.470 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:19:28.470 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:32.658 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:32.650+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:19:33.866 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:19:33.867 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:37.662 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:37.654+0000 7f47c65300c0 1 waiting for incremental sync to catch up: 2020-11-19T03:19:37.663 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:19:37.663 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:42.665 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:42.658+0000 7f47c65300c0 1 bucket sync caught up with source: 2020-11-19T03:19:42.666 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:42.666 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.44.5, , , , , ] 2020-11-19T03:19:42.666 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:42.658+0000 7f47c65300c0 0 bucket checkpoint complete 2020-11-19T03:19:42.699 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-65 zones={test-zone2, test-zone1} 2020-11-19T03:19:42.715 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:19:42.762 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-19T03:19:42.763 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:19:42.771 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-19T03:19:42.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-19T03:19:43.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-19T03:19:43.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-19T03:19:44.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-65 zones={test-zone2, test-zone1} 2020-11-19T03:19:44.112 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-65 zones={test-zone3, test-zone1} 2020-11-19T03:19:44.134 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:19:44.188 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-19T03:19:44.189 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:19:44.201 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-19T03:19:44.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-19T03:19:44.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-19T03:19:44.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-19T03:19:45.106 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-65 zones={test-zone3, test-zone1} 2020-11-19T03:19:45.107 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-65 zones={test-zone3, test-zone2} 2020-11-19T03:19:45.119 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:19:45.171 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-19T03:19:45.171 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:19:45.179 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-19T03:19:45.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-19T03:19:45.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-19T03:19:45.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-19T03:19:46.066 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-65 zones={test-zone3, test-zone2} 2020-11-19T03:19:46.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2020-11-19T03:19:46.067 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-68 2020-11-19T03:19:46.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:46.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:46.207 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:46.241 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:46.242 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:46.486 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:19:46.526 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:19:46.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:46.527 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755986.107706_973.1", 2020-11-19T03:19:46.527 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:19:46.107706Z" 2020-11-19T03:19:46.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:46.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:46.527 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:46.528 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:19:46.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:46.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:46.528 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:46.529 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:19:46.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:46.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:46.529 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:46.529 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:19:46.530 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:46.530 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:19:46.531 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:19:46.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:46.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:46.532 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:46.575 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:46.575 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:46.824 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:19:46.824 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:19:46.825 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:19:46.825 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:19:46.825 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:19:46.825 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:19:46.826 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:46.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:46.826 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:19:46.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:46.827 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:19:46.827 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:46.827 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:46.828 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755986.107706_973.1", 2020-11-19T03:19:46.828 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:46.828 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:19:46.828 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:46.828 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:19:46.107706Z", 2020-11-19T03:19:46.829 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:46.829 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:46.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:46.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:46.830 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:19:46.830 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:46.830 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:46.830 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:19:46.830 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:46.831 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:46.831 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:46.831 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:19:46.831 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:46.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:46.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:46.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:46.832 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:19:46.833 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:46.833 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:46.833 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:46.833 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:46.834 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:46.834 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:46.834 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:46.835 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:46.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:46.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:46.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:46.836 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:19:46.836 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:46.836 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:46.836 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:46.837 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:46.837 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:46.837 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:46.837 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:46.838 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:46.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:46.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:46.839 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:19:46.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:46.839 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:19:46.839 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:19:46.840 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:19:46.840 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:46.840 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:19:46.841 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:19:46.842 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:19:46.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:46.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:46.843 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:46.886 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:46.887 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:47.169 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:19:47.170 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:19:47.170 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:19:47.170 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:19:47.171 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:19:47.171 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:19:47.171 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:19:47.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:47.171 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:19:47.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:47.172 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:19:47.172 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:47.172 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:47.173 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755986.107706_973.1", 2020-11-19T03:19:47.173 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:47.173 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:19:47.173 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:47.174 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:19:46.107706Z", 2020-11-19T03:19:47.174 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:19:47.174 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:47.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:47.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:47.175 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:19:47.176 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:47.176 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:47.176 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:19:47.176 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:47.177 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:19:47.177 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:47.177 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:19:47.177 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:19:47.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:47.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:47.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:47.179 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:19:47.179 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:47.179 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:47.180 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:47.180 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:47.180 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:19:47.180 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:47.181 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:19:47.181 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:19:47.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:47.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:47.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:47.182 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:19:47.182 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:47.182 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:47.182 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:47.183 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:47.183 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:19:47.183 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:47.183 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:19:47.184 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:19:47.184 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:47.184 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:47.184 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:19:47.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:47.185 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:19:47.185 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:19:47.185 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:19:47.185 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:47.186 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:19:47.187 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:19:47.187 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:19:47.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:47.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:47.191 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:47.233 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:47.234 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:47.502 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:19:47.502 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:19:47.502 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:19:47.503 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:19:47.503 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:19:47.503 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:19:47.503 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:47.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:47.504 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:19:47.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:47.504 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:19:47.504 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:47.504 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:47.505 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755986.107706_973.1", 2020-11-19T03:19:47.505 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:47.505 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:19:47.505 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:47.506 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:19:46.107706Z", 2020-11-19T03:19:47.506 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:47.506 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:47.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:47.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:47.507 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:19:47.507 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:47.507 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:47.507 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:19:47.507 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:47.508 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:47.508 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:47.508 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:19:47.508 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:47.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:47.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:47.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:47.509 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:19:47.509 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:47.509 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:47.510 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:47.510 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:47.510 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:47.510 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:47.510 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:47.511 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:47.511 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:47.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:47.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:47.511 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:19:47.512 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:47.512 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:47.512 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:47.512 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:47.512 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:19:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:19:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:19:47.515 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:19:47.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:47.515 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:19:47.516 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:19:47.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:47.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:47.518 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:47.606 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:47.606 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:47.859 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:47.853+0000 7f2d06c990c0 1 bucket sync caught up with source: 2020-11-19T03:19:47.859 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.62.5, , , , 00000000001.73.5, , , , , ] 2020-11-19T03:19:47.859 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.62.5, , , , 00000000001.73.5, , , , , ] 2020-11-19T03:19:47.859 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:47.853+0000 7f2d06c990c0 0 bucket checkpoint complete 2020-11-19T03:19:47.998 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2020-11-19T03:19:47.999 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-69 2020-11-19T03:19:48.068 INFO:tasks.rgw.c1.client.1.smithi086.stdout:2020-11-19T03:19:48.062+0000 7f97817fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:19:48.069 INFO:tasks.rgw.c2.client.1.smithi160.stdout:2020-11-19T03:19:48.061+0000 7f69b4ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:19:48.069 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2020-11-19T03:19:48.061+0000 7ff40b7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-19T03:19:48.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:48.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:48.243 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:48.290 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:48.386 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:48.513 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:19:48.536 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:19:48.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:48.537 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755988.038743_980.1", 2020-11-19T03:19:48.537 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:19:48.038743Z" 2020-11-19T03:19:48.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:48.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:48.538 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:48.538 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:19:48.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:48.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:48.539 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:48.539 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:19:48.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:48.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:48.539 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:48.540 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:19:48.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:48.540 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:19:48.540 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:19:48.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:48.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:48.542 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:48.586 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:48.587 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:48.824 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:19:48.827 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:19:48.828 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:19:48.829 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:19:48.829 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:19:48.830 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:19:48.830 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:48.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:48.832 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:19:48.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:48.833 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:19:48.834 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:48.835 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:48.836 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755988.038743_980.1", 2020-11-19T03:19:48.837 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:48.838 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:19:48.838 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:48.839 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:19:48.038743Z", 2020-11-19T03:19:48.839 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:48.840 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:48.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:48.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:48.842 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:19:48.842 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:48.842 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:48.843 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:19:48.843 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:48.843 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:48.844 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:48.844 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:19:48.845 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:48.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:48.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:48.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:48.847 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:19:48.847 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:48.848 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:48.848 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:48.848 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:48.849 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:48.849 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:48.849 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:48.850 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:48.851 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:48.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:48.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:48.851 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:19:48.851 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:48.852 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:48.852 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:48.852 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:48.852 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:48.853 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:48.853 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:48.853 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:48.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:48.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:48.854 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:19:48.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:48.854 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:19:48.854 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:19:48.854 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:19:48.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:48.855 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:19:48.856 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:19:48.856 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:19:48.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:48.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:48.858 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:48.900 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:48.901 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:49.174 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:19:49.174 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:19:49.175 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:19:49.175 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:19:49.175 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:19:49.175 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:19:49.176 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:19:49.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:49.176 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:19:49.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:49.176 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:19:49.177 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:49.177 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:49.177 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755988.038743_980.1", 2020-11-19T03:19:49.177 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:49.177 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:19:49.178 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:49.178 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:19:48.038743Z", 2020-11-19T03:19:49.178 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:19:49.179 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:49.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:49.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:49.179 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:19:49.180 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:49.180 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:49.180 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:19:49.180 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:49.181 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:19:49.181 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:49.181 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:19:49.182 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:19:49.182 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:49.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:49.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:49.182 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:19:49.183 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:49.183 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:49.183 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:49.183 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:49.183 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:19:49.184 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:49.184 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:19:49.184 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:19:49.184 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:49.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:49.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:49.185 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:19:49.185 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:49.185 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:49.186 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:49.186 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:49.186 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:19:49.186 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:49.186 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:19:49.187 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:19:49.187 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:49.187 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:49.187 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:19:49.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:49.188 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:19:49.188 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:19:49.188 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:19:49.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:49.188 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:19:49.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:19:49.190 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:19:49.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:49.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:49.192 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:49.240 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:49.241 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:49.510 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:19:49.511 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:19:49.511 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:19:49.512 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:19:49.512 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:19:49.512 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:19:49.513 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:49.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:49.514 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:19:49.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:49.514 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:19:49.515 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:49.515 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:49.516 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755988.038743_980.1", 2020-11-19T03:19:49.516 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:49.516 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:19:49.517 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:49.517 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:19:48.038743Z", 2020-11-19T03:19:49.518 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:49.518 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:49.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:49.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:49.519 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:19:49.520 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:49.520 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:49.520 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:19:49.521 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:49.521 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:49.522 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:49.522 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:19:49.523 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:49.523 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:49.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:49.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:49.524 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:19:49.524 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:49.524 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:49.525 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:49.525 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:49.525 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:49.526 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:49.526 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:49.526 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:49.527 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:49.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:49.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:49.528 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:19:49.528 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:49.528 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:49.528 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:49.528 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:49.529 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:49.529 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:49.529 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:49.529 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:49.529 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:49.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:49.530 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:19:49.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:49.530 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:19:49.531 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:19:49.531 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:19:49.531 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:49.532 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:19:49.534 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:19:49.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:49.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:49.536 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:49.631 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:49.631 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:49.902 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:49.893+0000 7fce859d30c0 1 bucket sync caught up with source: 2020-11-19T03:19:49.902 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.57.5, , , , 00000000001.52.5, , , 00000000001.68.5, , 00000000001.65.5] 2020-11-19T03:19:49.902 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.57.5, , , , 00000000001.52.5, , , 00000000001.68.5, , 00000000001.65.5] 2020-11-19T03:19:49.902 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:49.893+0000 7fce859d30c0 0 bucket checkpoint complete 2020-11-19T03:19:49.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:49.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:49.934 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:49.982 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:49.983 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:50.217 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:50.210+0000 7f4ab4fd70c0 1 bucket sync caught up with source: 2020-11-19T03:19:50.218 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.57.5, , , , 00000000001.52.5, , , 00000000001.68.5, , 00000000001.65.5] 2020-11-19T03:19:50.218 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.57.5, , , , 00000000001.52.5, , , 00000000001.68.5, , 00000000001.65.5] 2020-11-19T03:19:50.218 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:50.210+0000 7f4ab4fd70c0 0 bucket checkpoint complete 2020-11-19T03:19:50.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:50.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:50.248 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:50.296 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:50.296 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:50.558 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:50.553+0000 7fac21d460c0 1 bucket sync caught up with source: 2020-11-19T03:19:50.559 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.57.5, , , , 00000000001.52.5, , , 00000000001.68.5, , 00000000001.65.5] 2020-11-19T03:19:50.559 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.57.5, , , , 00000000001.52.5, , , 00000000001.68.5, , 00000000001.65.5] 2020-11-19T03:19:50.559 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:50.553+0000 7fac21d460c0 0 bucket checkpoint complete 2020-11-19T03:19:50.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:50.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:50.673 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:50.721 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:50.721 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:50.988 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:50.982+0000 7f27a92830c0 1 bucket sync caught up with source: 2020-11-19T03:19:50.988 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.55.5, , , , 00000000001.46.5, , , 00000000001.90.5, , 00000000001.62.5] 2020-11-19T03:19:50.988 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.55.5, , , , 00000000001.46.5, , , 00000000001.90.5, , 00000000001.62.5] 2020-11-19T03:19:50.989 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:50.982+0000 7f27a92830c0 0 bucket checkpoint complete 2020-11-19T03:19:51.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:51.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:51.022 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:51.074 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:51.074 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:51.321 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:51.314+0000 7fbb109250c0 1 bucket sync caught up with source: 2020-11-19T03:19:51.322 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.55.5, , , , 00000000001.46.5, , , 00000000001.90.5, , 00000000001.62.5] 2020-11-19T03:19:51.322 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.55.5, , , , 00000000001.46.5, , , 00000000001.90.5, , 00000000001.62.5] 2020-11-19T03:19:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:51.314+0000 7fbb109250c0 0 bucket checkpoint complete 2020-11-19T03:19:51.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:51.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:51.353 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:51.401 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:51.401 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:51.658 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:51.654+0000 7fe35632a0c0 1 bucket sync caught up with source: 2020-11-19T03:19:51.659 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.57.5, , , , 00000000001.48.5, , , 00000000001.70.5, , 00000000001.70.5] 2020-11-19T03:19:51.659 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.57.5, , , , 00000000001.48.5, , , 00000000001.70.5, , 00000000001.70.5] 2020-11-19T03:19:51.659 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:51.654+0000 7fe35632a0c0 0 bucket checkpoint complete 2020-11-19T03:19:51.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:51.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:51.689 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:51.738 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:51.738 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:52.002 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:51.997+0000 7fd8524770c0 1 bucket sync caught up with source: 2020-11-19T03:19:52.003 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.57.5, , , , 00000000001.48.5, , , 00000000001.70.5, , 00000000001.70.5] 2020-11-19T03:19:52.003 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.57.5, , , , 00000000001.48.5, , , 00000000001.70.5, , 00000000001.70.5] 2020-11-19T03:19:52.003 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:51.997+0000 7fd8524770c0 0 bucket checkpoint complete 2020-11-19T03:19:52.034 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-69 zones={test-zone2, test-zone1} 2020-11-19T03:19:52.046 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:19:52.098 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:19:52.099 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:19:52.099 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:19:52.100 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:19:52.100 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:19:52.108 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:19:52.109 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:19:52.109 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:19:52.110 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:19:52.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:19:52.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:19:52.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:19:52.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:19:52.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:19:52.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:19:52.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:19:52.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:19:52.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:19:53.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:19:53.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:19:53.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:19:53.384 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-69 zones={test-zone2, test-zone1} 2020-11-19T03:19:53.385 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-69 zones={test-zone3, test-zone1} 2020-11-19T03:19:53.394 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:19:53.399 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:19:53.400 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:19:53.400 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:19:53.401 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:19:53.401 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:19:53.406 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:19:53.407 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:19:53.407 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:19:53.408 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:19:53.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:19:53.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:19:53.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:19:53.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:19:53.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:19:53.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:19:53.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:19:54.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:19:54.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:19:54.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:19:54.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:19:54.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:19:54.670 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-69 zones={test-zone3, test-zone1} 2020-11-19T03:19:54.671 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-69 zones={test-zone3, test-zone2} 2020-11-19T03:19:54.688 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:19:54.703 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:19:54.704 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:19:54.705 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:19:54.706 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:19:54.706 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:19:54.725 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:19:54.725 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:19:54.726 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:19:54.726 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:19:54.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:19:54.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:19:54.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:19:55.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:19:55.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:19:55.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:19:55.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:19:55.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:19:55.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:19:55.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:19:55.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:19:55.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:19:56.022 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-69 zones={test-zone3, test-zone2} 2020-11-19T03:19:56.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:19:56.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:19:56.024 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:19:56.062 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:56.063 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:56.344 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=atsuux-70 2020-11-19T03:19:56.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:56.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:56.583 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:56.616 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:56.617 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:19:56.856 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:19:56.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:56.856 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:19:56.857 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:19:56.376123Z" 2020-11-19T03:19:56.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:56.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:56.857 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:56.858 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:19:56.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:56.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:56.858 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:56.858 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:19:56.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:56.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:56.859 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:56.859 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:19:56.859 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:56.860 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:19:56.861 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:19:56.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:56.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:56.863 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:56.907 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:56.907 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:57.156 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:19:57.157 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:19:57.157 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:19:57.157 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:19:57.158 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:19:57.158 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:19:57.158 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:57.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.158 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:19:57.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:57.159 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:19:57.159 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:57.160 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:57.160 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:19:57.160 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:57.160 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:19:57.160 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:57.161 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:19:56.376123Z", 2020-11-19T03:19:57.161 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:57.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:57.162 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:19:57.162 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:57.162 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:57.162 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:19:57.163 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:57.163 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:57.163 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:57.163 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:19:57.164 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:57.164 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:57.164 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:19:57.165 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:57.165 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:57.165 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:57.165 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:57.165 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:57.166 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:57.166 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:57.166 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:57.166 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:57.167 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:19:57.167 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:57.167 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:57.167 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:57.168 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:57.168 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:57.168 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:57.168 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:57.169 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:57.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.169 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:19:57.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.170 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:19:57.170 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:19:57.171 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:19:57.171 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.171 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:19:57.172 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:19:57.173 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:19:57.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:57.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:57.174 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:57.214 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:57.215 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:57.472 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:19:57.473 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:19:57.473 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:19:57.473 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:19:57.474 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:19:57.474 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:19:57.474 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:19:57.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:57.474 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:19:57.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:57.475 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:19:57.475 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:57.475 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:57.476 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:19:57.476 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:57.476 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:19:57.476 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:57.476 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:19:56.376123Z", 2020-11-19T03:19:57.477 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:19:57.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:57.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:57.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:57.477 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:19:57.478 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:57.478 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:57.478 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:19:57.478 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:57.478 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:19:57.479 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:57.479 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:19:57.479 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:19:57.479 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:57.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:57.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:57.480 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:19:57.480 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:57.480 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:57.481 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:57.481 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:57.481 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:19:57.481 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:57.481 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:19:57.482 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:19:57.482 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:57.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:57.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:19:57.482 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:19:57.483 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:19:57.483 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:19:57.483 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:19:57.483 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:19:57.483 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:19:57.484 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:19:57.484 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:19:57.484 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:19:57.484 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:57.484 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:57.485 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:19:57.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:19:57.485 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:19:57.485 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:19:57.485 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:19:57.486 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:19:57.486 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:19:57.487 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:19:57.488 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:19:57.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:57.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:19:57.489 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:19:57.530 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:57.530 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:57.791 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:19:57.792 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:19:57.792 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:19:57.792 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:19:57.792 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:19:57.793 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:19:57.793 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:57.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.793 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:19:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:19:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:57.794 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:19:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:19:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:19:56.376123Z", 2020-11-19T03:19:57.795 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:19:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:19:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:57.797 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:57.797 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:19:57.797 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:57.797 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:19:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:19:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:57.799 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:57.800 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:57.800 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:57.800 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:57.800 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:57.800 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:19:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:19:57.801 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:19:57.802 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:19:57.802 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:19:57.802 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:19:57.802 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:19:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:19:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:19:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:19:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.803 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:19:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:19:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:19:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:19:57.804 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:19:57.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:19:57.805 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:19:57.806 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:19:57.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:57.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:57.807 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:57.897 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:57.897 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:58.147 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:58.141+0000 7fa84b9340c0 1 bucket sync caught up with source: 2020-11-19T03:19:58.147 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.59.5, , , , 00000000001.45.5, , , 00000000001.74.5, , 00000000001.47.5] 2020-11-19T03:19:58.148 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.45.5, , , 00000000001.74.5, , 00000000001.47.5] 2020-11-19T03:19:58.148 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:58.141+0000 7fa84b9340c0 0 bucket checkpoint complete 2020-11-19T03:19:58.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:58.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:58.182 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:58.228 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:58.229 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:58.489 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:58.482+0000 7f58dd05f0c0 1 bucket sync caught up with source: 2020-11-19T03:19:58.489 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.59.5, , , , 00000000001.45.5, , , 00000000001.74.5, , 00000000001.47.5] 2020-11-19T03:19:58.490 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.45.5, , , 00000000001.74.5, , 00000000001.47.5] 2020-11-19T03:19:58.490 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:58.482+0000 7f58dd05f0c0 0 bucket checkpoint complete 2020-11-19T03:19:58.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:58.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:58.519 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:58.566 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:58.567 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:58.801 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:58.793+0000 7f935b7230c0 1 bucket sync caught up with source: 2020-11-19T03:19:58.801 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.59.5, , , , 00000000001.45.5, , , 00000000001.74.5, , 00000000001.47.5] 2020-11-19T03:19:58.801 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.45.5, , , 00000000001.74.5, , 00000000001.47.5] 2020-11-19T03:19:58.802 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:19:58.793+0000 7f935b7230c0 0 bucket checkpoint complete 2020-11-19T03:19:58.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:58.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:58.934 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:58.982 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:58.983 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:59.254 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:59.250+0000 7f334ab560c0 1 bucket sync caught up with source: 2020-11-19T03:19:59.255 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.47.5, , , , 00000000001.41.5, , , 00000000001.63.5, , 00000000001.58.5] 2020-11-19T03:19:59.255 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.47.5, , , , 00000000001.41.5, , , 00000000001.63.5, , 00000000001.58.5] 2020-11-19T03:19:59.255 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:59.250+0000 7f334ab560c0 0 bucket checkpoint complete 2020-11-19T03:19:59.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:59.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:59.291 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:59.336 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:59.592 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:59.586+0000 7f4949f760c0 1 bucket sync caught up with source: 2020-11-19T03:19:59.593 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.47.5, , , , 00000000001.41.5, , , 00000000001.63.5, , 00000000001.58.5] 2020-11-19T03:19:59.593 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.47.5, , , , 00000000001.41.5, , , 00000000001.63.5, , 00000000001.58.5] 2020-11-19T03:19:59.593 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:59.586+0000 7f4949f760c0 0 bucket checkpoint complete 2020-11-19T03:19:59.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:59.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:19:59.627 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket atsuux-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:19:59.714 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:19:59.715 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:19:59.980 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:59.974+0000 7f21785c80c0 1 bucket sync caught up with source: 2020-11-19T03:19:59.981 INFO:teuthology.orchestra.run.smithi086.stderr: local status: [, 00000000001.41.5, , , , 00000000001.41.5, , , 00000000001.62.5, , 00000000001.59.5] 2020-11-19T03:19:59.982 INFO:teuthology.orchestra.run.smithi086.stderr: remote markers: [, 00000000001.41.5, , , , 00000000001.41.5, , , 00000000001.62.5, , 00000000001.59.5] 2020-11-19T03:19:59.982 INFO:teuthology.orchestra.run.smithi086.stderr:2020-11-19T03:19:59.974+0000 7f21785c80c0 0 bucket checkpoint complete 2020-11-19T03:20:00.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:20:00.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:20:00.017 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:20:00.068 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:00.082 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:00.300 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:20:00.293+0000 7f7c5355a0c0 1 bucket sync caught up with source: 2020-11-19T03:20:00.301 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.41.5, , , , 00000000001.41.5, , , 00000000001.62.5, , 00000000001.59.5] 2020-11-19T03:20:00.302 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.41.5, , , , 00000000001.41.5, , , 00000000001.62.5, , 00000000001.59.5] 2020-11-19T03:20:00.302 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:20:00.293+0000 7f7c5355a0c0 0 bucket checkpoint complete 2020-11-19T03:20:00.334 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-70 zones={test-zone2, test-zone1} 2020-11-19T03:20:00.360 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:20:00.376 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:20:00.376 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:20:00.377 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:20:00.377 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:20:00.378 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:20:00.387 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:20:00.388 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:20:00.388 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:20:00.389 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:20:00.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:20:00.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:20:00.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:20:00.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:20:00.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:20:00.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:20:01.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:20:01.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:20:01.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:20:01.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:20:01.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:20:01.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:20:01.656 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-70 zones={test-zone2, test-zone1} 2020-11-19T03:20:01.656 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-70 zones={test-zone3, test-zone1} 2020-11-19T03:20:01.667 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:20:01.673 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:20:01.674 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:20:01.675 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:20:01.675 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:20:01.676 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:20:01.685 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:20:01.686 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:20:01.687 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:20:01.688 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:20:01.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:20:01.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:20:01.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:20:01.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:20:02.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:20:02.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:20:02.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:20:02.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:20:02.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:20:02.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:20:02.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:20:02.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:20:02.825 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-70 zones={test-zone3, test-zone1} 2020-11-19T03:20:02.826 INFO:tasks.rgw_multi.zone_rados:comparing bucket=atsuux-70 zones={test-zone3, test-zone2} 2020-11-19T03:20:02.838 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:20:02.843 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:20:02.843 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:20:02.844 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:20:02.844 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:20:02.845 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:20:02.852 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-19T03:20:02.852 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-19T03:20:02.853 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-19T03:20:02.853 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-19T03:20:02.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:20:02.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:20:03.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-19T03:20:03.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:20:03.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:20:03.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-19T03:20:03.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:20:03.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:20:03.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-19T03:20:03.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:20:03.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:20:03.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-19T03:20:03.974 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=atsuux-70 zones={test-zone3, test-zone2} 2020-11-19T03:20:03.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:20:03.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:20:03.976 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:20:04.014 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:04.015 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:04.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'atsuux-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:04.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'atsuux-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:04.327 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket atsuux-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:04.405 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:04.406 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T03:20:04.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'atsuux-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:04.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'atsuux-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:04.700 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket atsuux-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:05.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:05.056 INFO:teuthology.orchestra.run.smithi086.stdout: "op_id": "1#00000000001.57.5", 2020-11-19T03:20:05.056 INFO:teuthology.orchestra.run.smithi086.stdout: "op_tag": "d71482d6-15fd-4715-9992-a25958dcb538.5484.5404", 2020-11-19T03:20:05.056 INFO:teuthology.orchestra.run.smithi086.stdout: "op": "write", 2020-11-19T03:20:05.056 INFO:teuthology.orchestra.run.smithi086.stdout: "object": "a", 2020-11-19T03:20:05.057 INFO:teuthology.orchestra.run.smithi086.stdout: "instance": "", 2020-11-19T03:20:05.057 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "complete", 2020-11-19T03:20:05.057 INFO:teuthology.orchestra.run.smithi086.stdout: "index_ver": 1, 2020-11-19T03:20:05.057 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:19:48.050679915Z", 2020-11-19T03:20:05.057 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": { 2020-11-19T03:20:05.058 INFO:teuthology.orchestra.run.smithi086.stdout: "pool": 9, 2020-11-19T03:20:05.058 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 4 2020-11-19T03:20:05.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:05.058 INFO:teuthology.orchestra.run.smithi086.stdout: "bilog_flags": 0, 2020-11-19T03:20:05.058 INFO:teuthology.orchestra.run.smithi086.stdout: "versioned": false, 2020-11-19T03:20:05.059 INFO:teuthology.orchestra.run.smithi086.stdout: "owner": "", 2020-11-19T03:20:05.059 INFO:teuthology.orchestra.run.smithi086.stdout: "owner_display_name": "", 2020-11-19T03:20:05.059 INFO:teuthology.orchestra.run.smithi086.stdout: "zones_trace": [ 2020-11-19T03:20:05.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:05.059 INFO:teuthology.orchestra.run.smithi086.stdout: "entry": "d71482d6-15fd-4715-9992-a25958dcb538:atsuux-69:d71482d6-15fd-4715-9992-a25958dcb538.5486.20" 2020-11-19T03:20:05.060 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:05.060 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:05.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:05.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:05.061 INFO:teuthology.orchestra.run.smithi086.stdout: "op_id": "5#00000000001.52.5", 2020-11-19T03:20:05.061 INFO:teuthology.orchestra.run.smithi086.stdout: "op_tag": "d71482d6-15fd-4715-9992-a25958dcb538.5484.5425", 2020-11-19T03:20:05.061 INFO:teuthology.orchestra.run.smithi086.stdout: "op": "write", 2020-11-19T03:20:05.061 INFO:teuthology.orchestra.run.smithi086.stdout: "object": "d", 2020-11-19T03:20:05.061 INFO:teuthology.orchestra.run.smithi086.stdout: "instance": "", 2020-11-19T03:20:05.062 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "complete", 2020-11-19T03:20:05.062 INFO:teuthology.orchestra.run.smithi086.stdout: "index_ver": 1, 2020-11-19T03:20:05.062 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:19:48.199533614Z", 2020-11-19T03:20:05.062 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": { 2020-11-19T03:20:05.062 INFO:teuthology.orchestra.run.smithi086.stdout: "pool": 9, 2020-11-19T03:20:05.063 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 4 2020-11-19T03:20:05.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:05.063 INFO:teuthology.orchestra.run.smithi086.stdout: "bilog_flags": 0, 2020-11-19T03:20:05.063 INFO:teuthology.orchestra.run.smithi086.stdout: "versioned": false, 2020-11-19T03:20:05.063 INFO:teuthology.orchestra.run.smithi086.stdout: "owner": "", 2020-11-19T03:20:05.064 INFO:teuthology.orchestra.run.smithi086.stdout: "owner_display_name": "", 2020-11-19T03:20:05.064 INFO:teuthology.orchestra.run.smithi086.stdout: "zones_trace": [ 2020-11-19T03:20:05.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:05.065 INFO:teuthology.orchestra.run.smithi086.stdout: "entry": "d71482d6-15fd-4715-9992-a25958dcb538:atsuux-69:d71482d6-15fd-4715-9992-a25958dcb538.5486.20" 2020-11-19T03:20:05.065 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:05.065 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:05.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:05.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:05.066 INFO:teuthology.orchestra.run.smithi086.stdout: "op_id": "8#00000000001.68.5", 2020-11-19T03:20:05.066 INFO:teuthology.orchestra.run.smithi086.stdout: "op_tag": "d71482d6-15fd-4715-9992-a25958dcb538.5484.5417", 2020-11-19T03:20:05.070 INFO:teuthology.orchestra.run.smithi086.stdout: "op": "write", 2020-11-19T03:20:05.070 INFO:teuthology.orchestra.run.smithi086.stdout: "object": "c", 2020-11-19T03:20:05.070 INFO:teuthology.orchestra.run.smithi086.stdout: "instance": "", 2020-11-19T03:20:05.071 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "complete", 2020-11-19T03:20:05.071 INFO:teuthology.orchestra.run.smithi086.stdout: "index_ver": 1, 2020-11-19T03:20:05.071 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:19:48.151331983Z", 2020-11-19T03:20:05.071 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": { 2020-11-19T03:20:05.071 INFO:teuthology.orchestra.run.smithi086.stdout: "pool": 9, 2020-11-19T03:20:05.072 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 3 2020-11-19T03:20:05.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:05.072 INFO:teuthology.orchestra.run.smithi086.stdout: "bilog_flags": 0, 2020-11-19T03:20:05.072 INFO:teuthology.orchestra.run.smithi086.stdout: "versioned": false, 2020-11-19T03:20:05.072 INFO:teuthology.orchestra.run.smithi086.stdout: "owner": "", 2020-11-19T03:20:05.073 INFO:teuthology.orchestra.run.smithi086.stdout: "owner_display_name": "", 2020-11-19T03:20:05.073 INFO:teuthology.orchestra.run.smithi086.stdout: "zones_trace": [ 2020-11-19T03:20:05.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:05.074 INFO:teuthology.orchestra.run.smithi086.stdout: "entry": "d71482d6-15fd-4715-9992-a25958dcb538:atsuux-69:d71482d6-15fd-4715-9992-a25958dcb538.5486.20" 2020-11-19T03:20:05.074 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:05.074 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:05.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:05.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:05.075 INFO:teuthology.orchestra.run.smithi086.stdout: "op_id": "10#00000000001.65.5", 2020-11-19T03:20:05.075 INFO:teuthology.orchestra.run.smithi086.stdout: "op_tag": "d71482d6-15fd-4715-9992-a25958dcb538.5484.5412", 2020-11-19T03:20:05.075 INFO:teuthology.orchestra.run.smithi086.stdout: "op": "write", 2020-11-19T03:20:05.076 INFO:teuthology.orchestra.run.smithi086.stdout: "object": "b", 2020-11-19T03:20:05.076 INFO:teuthology.orchestra.run.smithi086.stdout: "instance": "", 2020-11-19T03:20:05.076 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "complete", 2020-11-19T03:20:05.076 INFO:teuthology.orchestra.run.smithi086.stdout: "index_ver": 1, 2020-11-19T03:20:05.076 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:19:48.101960121Z", 2020-11-19T03:20:05.077 INFO:teuthology.orchestra.run.smithi086.stdout: "ver": { 2020-11-19T03:20:05.077 INFO:teuthology.orchestra.run.smithi086.stdout: "pool": 9, 2020-11-19T03:20:05.077 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 6 2020-11-19T03:20:05.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:05.077 INFO:teuthology.orchestra.run.smithi086.stdout: "bilog_flags": 0, 2020-11-19T03:20:05.078 INFO:teuthology.orchestra.run.smithi086.stdout: "versioned": false, 2020-11-19T03:20:05.078 INFO:teuthology.orchestra.run.smithi086.stdout: "owner": "", 2020-11-19T03:20:05.078 INFO:teuthology.orchestra.run.smithi086.stdout: "owner_display_name": "", 2020-11-19T03:20:05.078 INFO:teuthology.orchestra.run.smithi086.stdout: "zones_trace": [ 2020-11-19T03:20:05.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:05.079 INFO:teuthology.orchestra.run.smithi086.stdout: "entry": "d71482d6-15fd-4715-9992-a25958dcb538:atsuux-69:d71482d6-15fd-4715-9992-a25958dcb538.5486.20" 2020-11-19T03:20:05.079 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:05.079 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:05.079 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:05.079 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:05.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:20:05.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:20:05.084 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:20:05.157 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:05.157 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:06.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'atsuux-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:06.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'atsuux-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:06.675 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket atsuux-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:06.758 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:06.759 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:07.041 INFO:teuthology.orchestra.run.smithi086.stdout:[] 2020-11-19T03:20:07.042 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2020-11-19T03:20:07.095 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-11-19T03:20:07.096 INFO:tasks.rgw_multisite_tests: 2020-11-19T03:20:07.098 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-19T03:20:07.098 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1475.726s 2020-11-19T03:20:07.099 INFO:tasks.rgw_multisite_tests: 2020-11-19T03:20:07.099 INFO:tasks.rgw_multisite_tests:OK 2020-11-19T03:20:07.103 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-11-19T03:20:07.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:07.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:07.109 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:07.149 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:07.149 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:07.405 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:07.440 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:07.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:07.441 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:20:07.441 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:19:56.376123Z" 2020-11-19T03:20:07.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:07.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:07.442 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:07.442 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:07.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:07.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:07.443 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:07.443 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:07.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:07.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:07.444 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:07.444 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:07.444 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:07.444 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:07.445 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:07.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:07.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:07.447 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:07.491 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:07.491 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:07.778 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:07.779 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:07.779 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:07.779 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:07.779 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:07.779 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:07.780 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:07.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:07.780 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:07.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:07.781 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:07.781 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:07.781 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:07.781 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:20:07.781 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:07.782 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:07.782 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:07.782 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:19:56.376123Z", 2020-11-19T03:20:07.782 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:07.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:07.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:07.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:07.783 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:07.783 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:07.783 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:07.784 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:07.784 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:07.784 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:07.784 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:07.784 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:07.785 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:07.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:07.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:07.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:07.785 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:07.786 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:07.786 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:07.786 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:07.786 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:07.786 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:07.787 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:07.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:07.787 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:07.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:07.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:07.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:07.788 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:07.788 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:07.788 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:07.788 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:07.789 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:07.789 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:07.789 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:07.789 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:07.790 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:07.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:07.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:07.790 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:07.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:07.791 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:07.791 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:07.791 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:07.791 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:07.791 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:07.792 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:07.793 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:07.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:07.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:07.795 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:07.836 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:07.836 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:08.108 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:08.108 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:08.108 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:08.109 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:08.109 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:08.109 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:08.109 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:08.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:08.110 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:08.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:08.110 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:08.110 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:08.111 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:08.111 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:20:08.111 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:08.111 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:08.111 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:08.112 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:19:56.376123Z", 2020-11-19T03:20:08.112 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:08.113 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:08.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:08.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:08.113 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:08.114 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:08.114 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:08.114 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:08.114 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:08.114 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:08.115 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:08.115 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:08.115 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:08.115 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:08.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:08.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:08.116 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:08.116 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:08.116 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:08.116 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:08.117 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:08.117 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:08.118 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:08.118 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:08.118 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:08.118 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:08.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:08.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:08.119 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:08.119 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:08.119 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:08.119 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:08.120 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:08.120 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:08.120 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:08.120 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:08.120 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:08.121 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:08.121 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:08.121 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:08.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:08.121 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:08.122 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:08.122 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:08.122 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:08.122 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:08.123 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:08.124 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:08.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:08.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:08.125 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:08.167 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:08.167 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:08.433 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:08.434 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:08.434 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:08.434 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:08.435 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:08.435 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:08.435 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:08.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:08.436 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:08.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:08.436 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:08.436 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:08.436 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:08.437 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:20:08.437 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:08.437 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:08.437 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:08.438 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:19:56.376123Z", 2020-11-19T03:20:08.438 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:08.438 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:08.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:08.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:08.439 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:08.439 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:08.439 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:08.439 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:08.440 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:08.440 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:08.440 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:08.440 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:08.441 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:08.441 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:08.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:08.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:08.442 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:08.442 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:08.442 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:08.442 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:08.443 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:08.443 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:08.443 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:08.443 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:08.443 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:08.444 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:08.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:08.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:08.444 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:08.445 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:08.445 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:08.445 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:08.445 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:08.446 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:08.446 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:08.446 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:08.446 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:08.447 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:08.451 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:08.451 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:08.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:08.452 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:08.452 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:08.452 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:08.452 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:08.452 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:08.454 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:08.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:08.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:08.455 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:08.494 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:08.494 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:08.767 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:08.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:08.768 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:20:08.768 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:19:56.376123Z" 2020-11-19T03:20:08.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:08.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:08.769 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:08.769 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:08.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:08.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:08.770 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:08.770 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:08.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:08.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:08.771 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:08.771 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:08.771 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:08.771 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:08.772 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:08.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:08.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:08.774 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:08.813 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:08.813 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:09.081 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605755996.376123_999.1", 2020-11-19T03:20:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:19:56.376123Z", 2020-11-19T03:20:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.096 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:09.097 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:09.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:09.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:09.147 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:09.426 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:09.461 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:09.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:09.462 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756009.140092_1033.1", 2020-11-19T03:20:09.462 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:09.140092Z" 2020-11-19T03:20:09.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:09.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:09.463 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:09.463 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:09.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:09.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:09.464 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:09.464 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:09.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:09.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:09.465 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:09.465 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:09.466 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:09.466 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:09.467 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:09.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:09.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:09.469 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:09.777 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:09.778 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:09.778 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:09.778 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:09.778 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:09.779 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:09.779 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:09.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.779 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:09.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:09.780 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:09.780 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:09.780 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:09.780 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756009.140092_1033.1", 2020-11-19T03:20:09.781 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:09.781 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:09.781 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:09.781 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:09.140092Z", 2020-11-19T03:20:09.781 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:09.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:09.782 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:09.783 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:09.783 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:09.783 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:09.783 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:09.783 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:09.784 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:09.784 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:09.784 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:09.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:09.785 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:09.785 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:09.785 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:09.785 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:09.786 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:09.786 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:09.786 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:09.786 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:09.786 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:09.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:09.787 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:09.787 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:09.788 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:09.788 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:09.788 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:09.788 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:09.788 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:09.789 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:09.789 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:09.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.789 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:09.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:09.790 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:09.790 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:09.790 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:09.791 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:09.791 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:09.792 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:10.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:10.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:10.100 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:10.142 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:10.142 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:10.387 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:10.424 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:10.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:10.425 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756009.140092_1033.1", 2020-11-19T03:20:10.425 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:09.140092Z" 2020-11-19T03:20:10.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:10.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:10.425 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:10.426 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:10.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:10.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:10.426 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:10.426 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:10.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:10.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:10.427 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:10.427 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:10.427 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:10.428 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:10.429 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:10.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:10.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:10.430 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:10.488 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:10.489 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:10.759 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:10.760 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:10.760 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:10.760 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:10.760 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:10.761 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:10.761 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:10.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:10.761 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:10.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:10.762 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:10.762 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:10.762 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:10.762 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756009.140092_1033.1", 2020-11-19T03:20:10.762 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:10.763 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:10.763 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:10.763 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:09.140092Z", 2020-11-19T03:20:10.763 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:10.763 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:10.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:10.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:10.764 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:10.764 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:10.764 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:10.765 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:10.765 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:10.765 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:10.765 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:10.766 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:10.766 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:10.766 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:10.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:10.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:10.767 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:10.767 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:10.767 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:10.767 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:10.767 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:10.768 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:10.768 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:10.768 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:10.768 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:10.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:10.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:10.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:10.770 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:10.770 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:10.770 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:10.771 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:10.771 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:10.771 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:10.771 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:10.771 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:10.772 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:10.772 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:10.772 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:10.772 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:10.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:10.773 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:10.773 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:10.773 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:10.773 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:10.773 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:10.775 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:10.926 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2020-11-19T03:20:10.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:10.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:10.928 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:10.967 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:10.967 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:11.183 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:11.227 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756010.914166_1041.1", 2020-11-19T03:20:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:10.914166Z" 2020-11-19T03:20:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:11.232 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:11.234 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:11.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:11.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:11.236 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:11.276 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:11.277 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:11.552 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:11.553 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:11.553 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:11.553 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:11.554 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:11.554 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:11.555 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:11.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:11.555 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:11.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:11.556 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:11.556 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:11.556 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:11.557 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756010.914166_1041.1", 2020-11-19T03:20:11.557 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:11.557 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:11.558 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:11.558 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:10.914166Z", 2020-11-19T03:20:11.558 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:11.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:11.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:11.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:11.560 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:11.560 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:11.560 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:11.561 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:11.561 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:11.561 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:11.562 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:11.562 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:11.562 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:11.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:11.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:11.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:11.564 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:11.564 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:11.564 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:11.565 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:11.565 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:11.565 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:11.566 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:11.566 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:11.566 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:11.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:11.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:11.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:11.568 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:11.568 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:11.569 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:11.569 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:11.569 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:11.570 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:11.570 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:11.570 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:11.571 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:11.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:11.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:11.572 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:11.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:11.572 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:11.572 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:11.573 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:11.573 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:11.573 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:11.575 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:11.576 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:11.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:11.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:11.578 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:11.622 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:11.623 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:11.900 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:13.372 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:13.373 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:13.373 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:13.373 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:13.373 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:13.374 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:13.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:13.374 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:13.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:13.375 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:13.375 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:13.375 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:13.375 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756010.914166_1041.1", 2020-11-19T03:20:13.375 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:13.376 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:13.376 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:13.376 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:20:10.914166Z", 2020-11-19T03:20:13.376 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:13.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:13.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:13.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:13.377 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:13.377 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:13.377 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:13.378 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:13.378 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:13.378 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:13.378 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:13.379 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:13.379 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:13.379 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:13.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:13.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:13.380 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:13.380 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:13.380 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:13.380 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:13.380 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:13.381 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:13.381 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:13.381 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:13.381 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:13.381 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:13.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:13.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:13.382 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:13.382 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:13.382 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:13.383 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:13.383 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:13.383 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:13.383 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:13.384 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:13.384 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:13.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:13.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:13.384 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:13.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:13.385 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:13.385 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:13.385 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:13.385 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:13.386 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:13.387 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:13.387 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:13.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:13.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:13.389 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:13.434 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:13.434 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:13.688 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756010.914166_1041.1", 2020-11-19T03:20:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:10.914166Z", 2020-11-19T03:20:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:13.704 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:13.705 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:13.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:13.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:13.707 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:13.748 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:13.748 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:14.005 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:14.039 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:14.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:14.040 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756010.914166_1041.1", 2020-11-19T03:20:14.040 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:10.914166Z" 2020-11-19T03:20:14.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:14.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:14.041 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:14.041 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:14.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:14.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:14.042 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:14.042 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:14.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:14.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:14.044 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:14.044 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:14.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:14.044 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:14.046 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:14.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:14.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:14.048 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:14.088 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:14.088 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:14.349 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:14.350 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:14.350 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:14.350 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:14.350 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:14.350 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:14.351 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:14.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:14.351 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:14.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:14.352 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:14.352 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:14.352 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:14.352 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756010.914166_1041.1", 2020-11-19T03:20:14.352 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:14.353 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:14.353 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:14.353 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:10.914166Z", 2020-11-19T03:20:14.353 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:14.353 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:14.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:14.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:14.354 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:14.354 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:14.355 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:14.356 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:14.356 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:14.357 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:14.357 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:14.357 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:14.357 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:14.358 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:14.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:14.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:14.358 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:14.358 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:14.359 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:14.359 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:14.359 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:14.359 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:14.359 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:14.360 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:14.360 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:14.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:14.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:14.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:14.361 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:14.361 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:14.361 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:14.361 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:14.362 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:14.362 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:14.362 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:14.362 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:14.362 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:14.363 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:14.363 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:14.363 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:14.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:14.363 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:14.364 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:14.364 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:14.364 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:14.364 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:14.365 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:14.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:14.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:14.416 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:14.689 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:14.727 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:14.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:14.728 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756014.406901_1051.1", 2020-11-19T03:20:14.728 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:14.406901Z" 2020-11-19T03:20:14.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:14.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:14.729 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:14.729 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:14.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:14.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:14.729 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:14.730 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:14.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:14.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:14.730 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:14.731 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:14.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:14.731 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:14.732 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:14.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:14.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:14.733 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:14.775 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:14.776 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:15.038 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:15.038 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:15.038 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:15.039 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:15.039 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:15.039 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:15.039 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:15.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.040 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:15.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:15.040 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:15.041 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:15.041 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:15.041 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756014.406901_1051.1", 2020-11-19T03:20:15.041 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:15.041 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:15.042 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:15.042 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:14.406901Z", 2020-11-19T03:20:15.042 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:15.042 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:15.043 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:15.043 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:15.044 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:15.044 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:15.044 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:15.044 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:15.045 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:15.045 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:15.045 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:15.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:15.046 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:15.046 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:15.046 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:15.047 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:15.047 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:15.047 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:15.047 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:15.047 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:15.048 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:15.048 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:15.049 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:15.049 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:15.049 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:15.049 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:15.050 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:15.050 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:15.051 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:15.051 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:15.051 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:15.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.052 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:15.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.052 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:15.052 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:15.052 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:15.053 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.053 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:15.056 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:15.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:15.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:15.112 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:15.153 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:15.153 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:15.418 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:15.452 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:15.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:15.453 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756014.406901_1051.1", 2020-11-19T03:20:15.453 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:14.406901Z" 2020-11-19T03:20:15.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:15.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:15.455 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:15.455 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:15.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:15.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:15.456 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:15.456 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:15.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:15.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:15.459 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:15.459 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:15.459 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:15.460 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:15.461 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:15.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:15.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:15.462 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:15.511 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:15.512 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:15.770 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756014.406901_1051.1", 2020-11-19T03:20:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:14.406901Z", 2020-11-19T03:20:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:15.776 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:15.776 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:15.776 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:15.776 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:15.776 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:15.777 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:15.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:15.777 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:15.778 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:15.778 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:15.778 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:15.778 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:15.778 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:15.779 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:15.779 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:15.779 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:15.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:15.780 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:15.780 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:15.780 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:15.780 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:15.781 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:15.781 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:15.781 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:15.781 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:15.781 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:15.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.782 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:15.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:15.782 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:15.783 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:15.783 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:15.783 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:15.783 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:15.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:16.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:20:16.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:20:16.248 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:20:16.306 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:16.306 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:16.517 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:20:16.509+0000 7fa4eb9720c0 1 waiting for incremental sync to catch up: 2020-11-19T03:20:16.517 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000001.85.5] 2020-11-19T03:20:16.517 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000002.87.5] 2020-11-19T03:20:21.521 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:20:21.513+0000 7fa4eb9720c0 1 waiting for incremental sync to catch up: 2020-11-19T03:20:21.522 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000001.85.5] 2020-11-19T03:20:21.522 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000002.87.5] 2020-11-19T03:20:26.526 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:20:26.517+0000 7fa4eb9720c0 1 waiting for incremental sync to catch up: 2020-11-19T03:20:26.526 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000001.85.5] 2020-11-19T03:20:26.526 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000002.87.5] 2020-11-19T03:20:31.530 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:20:31.521+0000 7fa4eb9720c0 1 waiting for incremental sync to catch up: 2020-11-19T03:20:31.530 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000001.85.5] 2020-11-19T03:20:31.530 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000002.87.5] 2020-11-19T03:20:36.534 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:20:36.529+0000 7fa4eb9720c0 1 bucket sync caught up with source: 2020-11-19T03:20:36.535 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000002.87.5] 2020-11-19T03:20:36.535 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.88.5, , 00000000001.83.5, 00000000001.65.5, 00000000001.79.5, , 00000000001.66.5, 00000000001.75.5, 00000000001.77.5, 00000000002.87.5] 2020-11-19T03:20:36.535 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:20:36.529+0000 7fa4eb9720c0 0 bucket checkpoint complete 2020-11-19T03:20:36.587 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:15.792513Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '0', 'size': 0, 'etag': '18886f92', 'versionId': '', 'sequencer': '6FE4B55FD45F9138', 'metadata': [], 'tags': []}}, 'eventId': '1605756015.949051.18886f92', 'opaqueData': ''} 2020-11-19T03:20:36.587 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:15.851806Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '2', 'size': 0, 'etag': '261b6dc5', 'versionId': '', 'sequencer': '6FE4B55F0ABD1F3B', 'metadata': [], 'tags': []}}, 'eventId': '1605756015.991935.261b6dc5', 'opaqueData': ''} 2020-11-19T03:20:36.588 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:15.806753Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '1', 'size': 0, 'etag': 'f25a4a46', 'versionId': '', 'sequencer': '6FE4B55F3CDA7E3B', 'metadata': [], 'tags': []}}, 'eventId': '1605756015.998169.f25a4a46', 'opaqueData': ''} 2020-11-19T03:20:36.588 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:15.895716Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '3', 'size': 0, 'etag': '6a58d325', 'versionId': '', 'sequencer': '70E4B55F11215908', 'metadata': [], 'tags': []}}, 'eventId': '1605756016.140058.6a58d325', 'opaqueData': ''} 2020-11-19T03:20:36.589 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:15.939804Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '4', 'size': 0, 'etag': '0ba5e7e0', 'versionId': '', 'sequencer': '70E4B55FFCEA6E0B', 'metadata': [], 'tags': []}}, 'eventId': '1605756016.191818.0ba5e7e0', 'opaqueData': ''} 2020-11-19T03:20:36.589 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:16.046231Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '6', 'size': 0, 'etag': '4a111456', 'versionId': '', 'sequencer': '70E4B55F0C5F720B', 'metadata': [], 'tags': []}}, 'eventId': '1605756016.192044.4a111456', 'opaqueData': ''} 2020-11-19T03:20:36.589 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:15.993521Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '5', 'size': 0, 'etag': 'ce264f55', 'versionId': '', 'sequencer': '70E4B55F070C790B', 'metadata': [], 'tags': []}}, 'eventId': '1605756016.192482.ce264f55', 'opaqueData': ''} 2020-11-19T03:20:36.590 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:16.152758Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '8', 'size': 0, 'etag': '5da57ba6', 'versionId': '', 'sequencer': '70E4B55FA47BFC0D', 'metadata': [], 'tags': []}}, 'eventId': '1605756016.234650.5da57ba6', 'opaqueData': ''} 2020-11-19T03:20:36.590 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:16.099595Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '7', 'size': 0, 'etag': 'c3d1fd50', 'versionId': '', 'sequencer': '70E4B55F418FE316', 'metadata': [], 'tags': []}}, 'eventId': '1605756016.384012.c3d1fd50', 'opaqueData': ''} 2020-11-19T03:20:36.591 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-19T03:20:16.201608Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'atsuux-72_notif', 'bucket': {'name': 'atsuux-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::atsuux-72', 'id': 'd71482d6-15fd-4715-9992-a25958dcb538.5486.23'}, 'object': {'key': '9', 'size': 0, 'etag': 'eb151e9d', 'versionId': '', 'sequencer': '80E4B55F46D6AC14', 'metadata': [], 'tags': []}}, 'eventId': '1605756032.346871.eb151e9d', 'opaqueData': ''} 2020-11-19T03:20:36.600 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2020-11-19T03:20:36.798 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2020-11-19T03:20:36.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:36.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:36.800 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:36.841 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:36.841 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:37.067 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:37.109 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:37.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:37.109 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756036.789210_1059.1", 2020-11-19T03:20:37.110 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:36.789210Z" 2020-11-19T03:20:37.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:37.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:37.110 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:37.110 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:37.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:37.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:37.111 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:37.111 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:37.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:37.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:37.112 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:37.112 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:37.112 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:37.113 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:37.113 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:37.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:37.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:37.114 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:37.154 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:37.155 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:37.424 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:37.424 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:37.425 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:37.425 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:37.425 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:37.425 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:37.425 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:37.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:37.426 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:37.426 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:37.426 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:37.428 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:37.428 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:37.428 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756036.789210_1059.1", 2020-11-19T03:20:37.428 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:37.429 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:37.429 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:37.429 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:36.789210Z", 2020-11-19T03:20:37.429 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:37.429 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:37.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:37.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:37.430 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:37.430 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:37.431 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:37.431 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:37.431 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:37.431 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:37.431 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:37.432 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:37.432 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:37.432 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:37.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:37.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:37.433 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:37.433 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:37.433 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:37.433 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:37.433 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:37.434 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:37.434 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:37.434 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:37.434 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:37.434 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:37.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:37.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:37.435 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:37.435 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:37.435 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:37.436 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:37.436 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:37.436 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:37.436 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:37.436 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:37.437 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:37.437 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:37.437 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:37.437 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:37.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:37.438 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:37.438 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:37.438 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:37.438 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:37.439 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:37.440 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:37.440 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:37.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:37.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:37.442 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:37.765 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:37.766 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:37.766 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:37.766 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:37.766 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:37.767 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:37.767 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:37.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:37.767 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:37.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:37.768 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:37.768 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:37.768 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:37.768 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756036.789210_1059.1", 2020-11-19T03:20:37.768 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:37.769 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:37.769 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:37.769 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:20:36.789210Z", 2020-11-19T03:20:37.769 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:37.770 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:37.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:37.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:37.770 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:37.770 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:37.771 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:37.771 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:37.771 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:37.771 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:37.771 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:37.772 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:37.772 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:37.772 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:37.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:37.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:37.773 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:37.773 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:37.773 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:37.773 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:37.773 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:37.774 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:37.774 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:37.774 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:37.774 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:37.774 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:37.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:37.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:37.775 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:37.775 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:37.775 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:37.776 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:37.776 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:37.776 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:37.776 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:37.776 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:37.777 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:37.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:37.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:37.777 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:37.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:37.778 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:37.778 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:37.778 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:37.778 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:37.778 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:37.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:37.780 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:37.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:37.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:37.782 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:37.827 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:37.827 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:38.098 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:38.098 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:38.098 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:38.099 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:38.099 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:38.099 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:38.099 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:38.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.100 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:38.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:38.100 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:38.100 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:38.101 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:38.101 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756036.789210_1059.1", 2020-11-19T03:20:38.101 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:38.101 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:38.102 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:38.102 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:36.789210Z", 2020-11-19T03:20:38.102 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:38.102 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:38.103 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:38.103 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:38.103 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:38.104 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:38.104 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:38.104 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:38.104 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:38.105 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:38.105 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:38.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:38.106 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:38.106 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:38.106 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:38.106 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:38.106 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:38.107 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:38.107 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:38.107 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:38.107 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:38.107 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:38.108 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:38.108 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:38.108 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:38.109 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:38.109 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:38.109 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:38.109 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:38.110 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:38.110 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:38.110 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.110 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.110 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:38.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.111 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:38.111 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:38.111 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:38.111 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.112 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:38.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:38.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:38.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:38.114 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:38.147 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:38.147 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:38.438 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:38.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:38.438 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756036.789210_1059.1", 2020-11-19T03:20:38.439 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:36.789210Z" 2020-11-19T03:20:38.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:38.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:38.439 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:38.440 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:38.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:38.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:38.440 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:38.440 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:38.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:38.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:38.441 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:38.441 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:38.442 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:38.442 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:38.443 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:38.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:38.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:38.444 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:38.499 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:38.500 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:38.770 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756036.789210_1059.1", 2020-11-19T03:20:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:36.789210Z", 2020-11-19T03:20:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:38.787 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:38.787 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:38.787 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:38.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:38.791 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:38.792 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:38.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:38.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:38.843 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:38.882 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:38.882 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:39.122 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:39.156 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:39.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:39.156 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756038.836362_1069.1", 2020-11-19T03:20:39.157 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:38.836362Z" 2020-11-19T03:20:39.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:39.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:39.157 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:39.158 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:39.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:39.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:39.158 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:39.158 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:39.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:39.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:39.160 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:39.160 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:39.161 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:39.161 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:39.162 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:39.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:39.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:39.164 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:39.203 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:39.204 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:39.471 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756038.836362_1069.1", 2020-11-19T03:20:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:38.836362Z", 2020-11-19T03:20:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:39.484 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:39.485 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:39.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:39.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:39.575 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:41.060 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:41.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:41.061 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756038.836362_1069.1", 2020-11-19T03:20:41.061 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:38.836362Z" 2020-11-19T03:20:41.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:41.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:41.062 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:41.062 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:41.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:41.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:41.063 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:41.063 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:41.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:41.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:41.064 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:41.065 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:41.065 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:41.065 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:41.066 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:41.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:41.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:41.069 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:41.117 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:41.118 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:41.375 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:41.375 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:41.376 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:41.376 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:41.376 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:41.376 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:41.377 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:41.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:41.377 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:41.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:41.377 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:41.378 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:41.378 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:41.378 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756038.836362_1069.1", 2020-11-19T03:20:41.378 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:41.379 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:41.380 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:41.380 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:38.836362Z", 2020-11-19T03:20:41.380 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:41.380 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:41.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:41.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:41.381 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:41.381 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:41.381 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:41.382 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:41.382 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:41.382 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:41.382 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:41.383 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:41.383 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:41.383 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:41.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:41.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:41.384 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:41.384 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:41.385 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:41.385 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:41.385 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:41.385 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:41.386 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:41.386 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:41.386 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:41.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:41.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:41.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:41.387 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:41.387 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:41.387 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:41.388 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:41.388 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:41.388 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:41.388 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:41.388 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:41.389 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:41.389 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:41.389 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:41.390 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:41.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:41.391 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:41.391 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:41.391 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:41.392 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:41.392 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:41.393 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:41.551 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2020-11-19T03:20:41.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:41.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:41.553 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:41.591 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:41.592 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:41.830 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:41.865 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:41.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:41.866 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:41.866 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:41.539869Z" 2020-11-19T03:20:41.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:41.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:41.867 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:41.867 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:41.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:41.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:41.867 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:41.868 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:41.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:41.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:41.868 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:41.868 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:41.869 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:41.869 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:41.870 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:41.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:41.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:41.871 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:41.913 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:41.913 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:42.196 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:42.196 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:42.196 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:42.197 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:42.197 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:42.197 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:42.197 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:42.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.198 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:42.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:42.198 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:42.198 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:42.198 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:42.199 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:42.199 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:42.199 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:42.199 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:42.200 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:42.200 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:42.200 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:42.201 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:42.201 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:42.201 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:42.201 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:42.202 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:42.202 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:42.202 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:42.202 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:42.202 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:42.203 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:42.203 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:42.203 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:42.204 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:42.204 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:42.204 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:42.204 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:42.204 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:42.205 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:42.205 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:42.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.206 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:42.206 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:42.206 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:42.206 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:42.206 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:42.207 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:42.207 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:42.207 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:42.207 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:42.207 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:42.208 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.208 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.208 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:42.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.208 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:42.209 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:42.209 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:42.209 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.209 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:42.210 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:42.211 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:42.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:42.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:42.212 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:42.251 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:42.252 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:42.530 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:42.530 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:42.531 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:42.531 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:42.531 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:42.531 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:42.531 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:42.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:42.532 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:42.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:42.532 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:42.532 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:42.533 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:42.533 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:42.533 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:42.533 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:42.534 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:42.534 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:42.534 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:42.534 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:42.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:42.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:42.535 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:42.535 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:42.535 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:42.535 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:42.536 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:42.536 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:42.537 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:42.537 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:42.537 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:42.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:42.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:42.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:42.538 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:42.538 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:42.539 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:42.539 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:42.539 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:42.539 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:42.539 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:42.540 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:42.541 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:42.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:42.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:42.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:42.541 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:42.542 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:42.542 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:42.542 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:42.542 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:42.542 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:42.543 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:42.543 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:42.543 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:42.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:42.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:42.544 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:42.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:42.544 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:42.544 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:42.544 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:42.545 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:42.546 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:42.547 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:42.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:42.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:42.549 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:42.588 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:42.588 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:42.952 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:42.953 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:42.953 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:42.953 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:42.954 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:42.954 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:42.954 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:42.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.954 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:42.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:42.955 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:42.955 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:42.955 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:42.955 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:42.956 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:42.956 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:42.956 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:42.956 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:42.957 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:42.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:42.957 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:42.958 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:42.958 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:42.958 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:42.958 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:42.958 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:42.959 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:42.959 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:42.959 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:42.959 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:42.960 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:42.960 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:42.960 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:42.960 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:42.961 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:42.961 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:42.961 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:42.961 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:42.961 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:42.962 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:42.962 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:42.963 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:42.963 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:42.963 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:42.963 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:42.963 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:42.964 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:42.964 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:42.964 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:42.964 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.964 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.965 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:42.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:42.965 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:42.965 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:42.965 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:42.966 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:42.966 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:42.967 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:42.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:42.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:42.969 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:43.009 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:43.010 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:43.233 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:41.539869Z" 2020-11-19T03:20:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:43.278 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:43.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:43.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:43.279 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:43.320 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:43.320 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:43.584 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:43.584 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:43.585 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:43.585 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:43.585 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:43.585 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:43.585 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:43.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:43.586 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:43.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:43.586 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:43.587 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:43.587 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:43.587 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:43.587 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:43.587 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:43.588 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:43.588 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:43.588 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:43.588 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:43.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:43.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:43.589 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:43.589 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:43.589 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:43.589 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:43.590 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:43.590 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:43.590 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:43.590 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:43.590 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:43.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:43.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:43.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:43.591 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:43.592 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:43.592 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:43.592 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:43.592 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:43.592 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:43.593 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:43.593 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:43.593 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:43.593 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:43.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:43.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:43.594 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:43.594 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:43.594 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:43.594 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:43.595 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:43.595 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:43.595 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:43.595 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:43.595 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:43.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:43.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:43.596 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:43.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:43.597 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:43.597 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:43.597 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:43.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:43.597 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:43.598 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:43.722 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2020-11-19T03:20:43.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:43.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:43.724 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:43.763 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:43.763 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:43.998 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:44.033 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:44.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:44.033 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:44.034 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:41.539869Z" 2020-11-19T03:20:44.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:44.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:44.034 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:44.034 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:44.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:44.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:44.035 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:44.035 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:44.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:44.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:44.036 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:44.036 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:44.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:44.036 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:44.037 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:44.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:44.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:44.039 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:44.095 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:44.096 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:44.361 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:44.362 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:44.362 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:44.362 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:44.363 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:44.363 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:44.363 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:44.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:44.363 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:44.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:44.364 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:44.364 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:44.364 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:44.364 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:44.365 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:44.365 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:44.365 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:44.365 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:44.365 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:44.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:44.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:44.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:44.366 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:44.366 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:44.367 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:44.367 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:44.367 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:44.367 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:44.367 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:44.368 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:44.368 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:44.368 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:44.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:44.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:44.369 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:44.369 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:44.369 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:44.369 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:44.369 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:44.370 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:44.370 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:44.370 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:44.370 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:44.370 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:44.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:44.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:44.371 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:44.371 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:44.371 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:44.372 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:44.372 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:44.372 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:44.372 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:44.372 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:44.373 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:44.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:44.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:44.373 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:44.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:44.374 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:44.374 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:44.374 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:44.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:44.374 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:44.377 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:44.378 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:44.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:44.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:44.379 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:44.419 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:44.419 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:44.708 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:44.708 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:44.708 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:44.709 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:44.709 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:44.709 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:44.709 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:44.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:44.710 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:44.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:44.710 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:44.711 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:44.712 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:44.712 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:44.712 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:44.712 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:44.713 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:44.714 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:44.715 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:44.715 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:44.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:44.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:44.715 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:44.716 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:44.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:44.718 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:44.719 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:44.720 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:44.720 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:44.720 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:44.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:44.721 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:44.721 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:44.721 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:44.721 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:44.722 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:44.723 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:44.723 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:44.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:44.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:44.725 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:44.765 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:44.766 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:45.134 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:45.147 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:45.147 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:45.147 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:45.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:45.147 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:45.148 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:45.148 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:45.148 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:45.148 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:45.149 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:45.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:45.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:45.151 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:45.202 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:45.202 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:45.455 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:47.058 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:47.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:47.059 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:47.059 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:41.539869Z" 2020-11-19T03:20:47.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:47.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:47.060 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:47.060 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:47.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:47.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:47.061 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:47.061 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:47.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:47.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:47.062 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:47.062 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:47.062 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:47.062 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:47.063 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:47.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:47.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:47.066 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:47.108 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:47.109 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:47.377 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:47.377 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:47.377 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:47.378 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:47.378 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:47.378 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:47.378 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:47.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:47.379 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:47.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:47.379 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:47.379 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:47.381 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:47.381 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:47.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:47.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:47.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:47.382 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:47.382 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:47.382 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:47.382 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:47.382 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:47.383 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:47.383 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:47.383 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:47.383 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:47.383 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:47.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:47.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:47.384 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:47.384 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:47.384 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:47.385 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:47.385 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:47.385 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:47.386 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:47.386 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:47.386 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:47.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:47.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:47.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:47.387 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:47.387 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:47.387 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:47.387 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:47.388 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:47.388 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:47.388 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:47.388 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:47.388 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:47.389 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:47.389 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:47.389 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:47.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:47.389 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:47.390 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:47.390 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:47.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:47.390 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:47.391 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:47.392 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2020-11-19T03:20:47.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:47.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:47.394 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:47.690 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:47.729 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:47.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:47.730 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:47.730 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:41.539869Z" 2020-11-19T03:20:47.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:47.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:47.731 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:47.731 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:47.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:47.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:47.732 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:47.732 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:47.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:47.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:47.733 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:47.733 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:47.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:47.733 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:47.734 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:47.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:47.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:47.736 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:47.775 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:47.776 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:48.062 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:48.078 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:48.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.078 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:48.080 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:48.080 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:48.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:48.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:48.082 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:48.124 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:48.124 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:48.399 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:48.400 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:48.400 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:48.400 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:48.400 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:48.401 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:48.401 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:48.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:48.401 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:48.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:48.402 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:48.402 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:48.403 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:48.403 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:48.403 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:48.403 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:48.403 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:48.404 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:48.404 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:48.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:48.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:48.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:48.405 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:48.406 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:48.407 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:48.408 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:48.408 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:48.408 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:48.408 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:48.409 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:48.409 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:48.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:48.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:48.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:48.410 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:48.410 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:48.410 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:48.411 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:48.411 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:48.411 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:48.411 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:48.411 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:48.412 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:48.412 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:48.412 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:48.412 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:48.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:48.413 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:48.413 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:48.413 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:48.413 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:48.414 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:48.415 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:48.415 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:48.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:48.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:48.417 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:48.456 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:48.457 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:48.723 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:48.724 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:48.724 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:48.724 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:48.724 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:48.724 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:48.725 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:48.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.725 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:48.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:48.726 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:48.726 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:48.726 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:48.726 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:48.726 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:48.727 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:48.727 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:48.727 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:48.727 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:48.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:48.728 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:48.728 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:48.728 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:48.729 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:48.729 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:48.729 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:48.729 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:48.729 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:48.730 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:48.730 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:48.730 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:48.731 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:48.731 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:48.731 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:48.731 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:48.731 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:48.732 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:48.732 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:48.732 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:48.732 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:48.733 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:48.733 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:48.733 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:48.734 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:48.734 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:48.734 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:48.734 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:48.734 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:48.735 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:48.735 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.735 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.735 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:48.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:48.736 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:48.736 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:48.736 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:48.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:48.736 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:48.737 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:48.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:48.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:48.739 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:48.779 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:49.021 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:49.056 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:49.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:49.057 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:49.057 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:41.539869Z" 2020-11-19T03:20:49.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:49.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:49.058 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:49.058 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:49.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:49.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:49.058 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:49.059 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:49.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:49.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:49.059 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:49.059 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:49.060 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:49.060 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:49.061 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:49.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:49.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:49.063 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:49.103 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:49.104 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:49.370 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:49.371 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:49.371 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:49.371 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:49.371 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:49.372 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:49.372 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:49.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:49.372 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:49.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:49.373 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:49.373 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:49.373 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:49.374 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756041.539869_1077.1", 2020-11-19T03:20:49.374 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:49.374 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:49.375 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:49.375 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:41.539869Z", 2020-11-19T03:20:49.375 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:49.375 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:49.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:49.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:49.376 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:49.376 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:49.376 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:49.376 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:49.377 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:49.377 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:49.377 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:49.377 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:49.377 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:49.378 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:49.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:49.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:49.378 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:49.378 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:49.379 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:49.379 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:49.379 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:49.379 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:49.379 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:49.380 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:49.380 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:49.380 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:49.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:49.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:49.381 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:49.381 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:49.381 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:49.381 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:49.381 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:49.382 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:49.382 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:49.382 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:49.382 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:49.383 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:49.383 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:49.383 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:49.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:49.383 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:49.383 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:49.384 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:49.384 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:49.384 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:49.385 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:49.685 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2020-11-19T03:20:49.686 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2020-11-19T03:20:49.687 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2020-11-19T03:20:49.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:49.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:49.689 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:49.730 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:49.731 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:49.971 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:50.011 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:50.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:50.011 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756049.676438_1101.1", 2020-11-19T03:20:50.012 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:49.676438Z" 2020-11-19T03:20:50.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:50.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:50.012 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:50.012 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:50.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:50.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:50.013 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:50.013 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:50.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:50.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:50.014 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:50.014 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:50.015 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:50.015 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:50.016 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:50.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:50.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:50.019 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:50.064 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:50.064 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:50.324 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:50.324 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:50.324 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:50.325 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:50.325 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:50.325 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:50.325 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:50.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:50.326 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:50.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:50.326 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:50.326 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:50.327 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:50.327 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756049.676438_1101.1", 2020-11-19T03:20:50.327 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:50.327 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:50.327 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:50.328 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:49.676438Z", 2020-11-19T03:20:50.328 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:50.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:50.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:50.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:50.329 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:50.329 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:50.330 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:50.330 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:50.330 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:50.330 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:50.330 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:50.331 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:50.331 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:50.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:50.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:50.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:50.332 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:50.332 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:50.332 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:50.332 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:50.332 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:50.333 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:50.333 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:50.333 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:50.333 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:50.333 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:50.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:50.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:50.334 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:50.334 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:50.334 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:50.335 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:50.335 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:50.335 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:50.335 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:50.335 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:50.336 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:50.336 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:50.336 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:50.336 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:50.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:50.337 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:50.337 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:50.337 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:50.337 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:50.338 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:50.339 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:50.339 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:50.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:50.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:50.341 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:50.383 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:50.384 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:50.674 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:50.675 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:50.675 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:50.675 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:50.676 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:50.676 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:50.677 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:50.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:50.677 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:50.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:50.678 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:50.678 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:50.678 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:50.679 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756049.676438_1101.1", 2020-11-19T03:20:50.679 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:50.679 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:50.679 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:50.679 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:20:49.676438Z", 2020-11-19T03:20:50.680 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:50.680 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:50.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:50.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:50.681 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:50.681 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:50.681 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:50.681 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:50.682 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:50.682 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:50.682 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:50.683 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:50.683 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:50.683 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:50.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:50.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:50.684 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:50.684 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:50.684 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:50.684 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:50.685 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:50.685 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:50.685 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:50.685 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:50.685 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:50.686 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:50.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:50.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:50.686 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:50.687 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:50.687 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:50.687 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:50.687 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:50.688 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:50.688 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:50.688 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:50.688 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:50.688 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:50.689 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:50.689 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:50.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:50.689 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:50.689 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:50.690 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:50.690 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:50.690 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:50.691 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:50.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:50.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:50.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:50.693 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:50.735 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:50.735 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:51.107 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:51.564 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:51.565 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:51.565 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:51.565 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:51.565 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:51.565 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:51.567 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:51.567 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756049.676438_1101.1", 2020-11-19T03:20:51.567 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:51.568 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:51.568 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:51.568 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:49.676438Z", 2020-11-19T03:20:51.568 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:51.568 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:51.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:51.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:51.569 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:51.569 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:51.570 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:51.570 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:51.570 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:51.574 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:51.574 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:51.574 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:51.574 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:51.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:51.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:51.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:51.575 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:51.575 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:51.576 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:51.576 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:51.576 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:51.576 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:51.576 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:51.577 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:51.577 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:51.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:51.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:51.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:51.578 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:51.578 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:51.578 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:51.578 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:51.578 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:51.580 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:51.580 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:51.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:51.580 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:51.580 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:51.581 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:51.581 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:51.581 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:51.582 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:51.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:51.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:51.585 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:51.878 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:51.918 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:51.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:51.919 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756049.676438_1101.1", 2020-11-19T03:20:51.920 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:49.676438Z" 2020-11-19T03:20:51.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:51.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:51.921 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:51.921 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:51.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:51.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:51.922 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:51.922 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:51.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:51.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:51.923 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:51.923 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:51.924 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:51.924 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:51.925 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:51.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:51.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:51.928 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:51.979 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:51.980 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:52.348 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:52.348 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:52.348 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:52.348 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:52.349 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:52.349 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:52.349 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:52.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:52.350 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:52.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:52.350 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:52.350 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:52.351 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:52.351 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756049.676438_1101.1", 2020-11-19T03:20:52.351 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:52.351 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:52.352 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:52.352 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:49.676438Z", 2020-11-19T03:20:52.352 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:52.352 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:52.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:52.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:52.353 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:52.353 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:52.353 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:52.354 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:52.354 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:52.354 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:52.354 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:52.355 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:52.355 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:52.355 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:52.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:52.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:52.356 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:52.356 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:52.356 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:52.356 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:52.357 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:52.357 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:52.357 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:52.357 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:52.357 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:52.358 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:52.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:52.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:52.358 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:52.359 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:52.359 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:52.359 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:52.359 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:52.359 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:52.360 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:52.360 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:52.360 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:52.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:52.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:52.361 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:52.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:52.361 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:52.361 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:52.361 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:52.362 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:52.362 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:52.363 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:52.643 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2020-11-19T03:20:52.646 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2020-11-19T03:20:52.648 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2020-11-19T03:20:52.648 INFO:tasks.rgw_multisite_tests:test reservation cleanup after gateway crash ... ok 2020-11-19T03:20:52.649 INFO:tasks.rgw_multisite_tests:test gateway recovery of persistent notifications ... ok 2020-11-19T03:20:52.650 INFO:tasks.rgw_multisite_tests:test pushing persistent notification via two gateways ... ok 2020-11-19T03:20:52.650 INFO:tasks.rgw_multisite_tests:test pushing persistent notification when one of the endpoints has error ... ok 2020-11-19T03:20:52.652 INFO:tasks.rgw_multisite_tests:test pushing persistent notification http ... ok 2020-11-19T03:20:52.652 INFO:tasks.rgw_multisite_tests:test pushing persistent notification amqp ... ok 2020-11-19T03:20:52.653 INFO:tasks.rgw_multisite_tests:test pushing persistent notification of large notifications ... ok 2020-11-19T03:20:52.653 INFO:tasks.rgw_multisite_tests:test pushing persistent notification pushback ... ok 2020-11-19T03:20:52.654 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-11-19T03:20:52.654 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-11-19T03:20:52.655 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2020-11-19T03:20:52.655 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2020-11-19T03:20:52.656 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2020-11-19T03:20:52.656 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2020-11-19T03:20:52.657 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2020-11-19T03:20:52.658 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2020-11-19T03:20:52.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:52.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:52.659 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:52.701 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:52.701 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:52.922 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:52.959 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:52.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:52.960 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:52.960 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:52.632067Z" 2020-11-19T03:20:52.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:52.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:52.961 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:52.961 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:52.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:52.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:52.962 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:52.962 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:52.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:52.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:52.963 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:52.963 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:52.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:52.964 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:52.965 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:52.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:52.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:52.967 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:53.009 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:53.010 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:53.282 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:53.282 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:53.283 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:53.283 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:53.283 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:53.283 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:53.283 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:53.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.284 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:53.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:53.284 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:53.285 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:53.285 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:53.285 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:53.285 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:53.285 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:53.286 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:53.286 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:53.286 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:53.286 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:53.287 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:53.287 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:53.287 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:53.287 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:53.288 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:53.288 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:53.288 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:53.288 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:53.288 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:53.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:53.289 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:53.289 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:53.290 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:53.290 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:53.290 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:53.290 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:53.290 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:53.291 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:53.291 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:53.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:53.292 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:53.292 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:53.292 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:53.292 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:53.292 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:53.293 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:53.293 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:53.293 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:53.293 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:53.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.294 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.294 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:53.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.294 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:53.294 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:53.295 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:53.295 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.295 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:53.296 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:53.297 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:53.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:53.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:53.298 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:53.340 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:53.340 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:53.621 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:53.622 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:53.622 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:53.622 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:53.623 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:53.623 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:53.623 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:53.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:53.623 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:53.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:53.624 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:53.624 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:53.624 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:53.624 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:53.625 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:53.625 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:53.625 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:53.626 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:53.626 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:53.626 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:53.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:53.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:53.627 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:53.627 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:53.627 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:53.627 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:53.628 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:53.628 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:53.628 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:53.628 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:53.628 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:53.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:53.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:53.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:53.629 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:53.629 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:53.630 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:53.630 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:53.630 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:53.630 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:53.630 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:53.631 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:53.631 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:53.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:53.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:53.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:53.632 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:53.632 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:53.632 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:53.632 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:53.632 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:53.633 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:53.633 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:53.633 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:53.633 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:53.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:53.634 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:53.634 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:53.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:53.634 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:53.634 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:53.635 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:53.635 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:53.636 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:53.637 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:53.637 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:53.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:53.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:53.639 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:53.681 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:53.682 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:53.946 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:53.947 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:53.947 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:53.947 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:53.947 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:53.947 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:53.948 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:53.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.948 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:53.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:53.949 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:53.949 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:53.949 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:53.949 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:53.949 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:53.950 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:53.950 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:53.950 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:53.950 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:53.951 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:53.951 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:53.951 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:53.952 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:53.952 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:53.952 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:53.952 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:53.952 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:53.954 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:53.954 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:53.954 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:53.954 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:53.954 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:53.955 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:53.955 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:53.955 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:53.955 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:53.955 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:53.956 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:53.956 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:53.956 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:53.957 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:53.957 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:53.957 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:53.957 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:53.957 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:53.958 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:53.958 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.958 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.958 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:53.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:53.959 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:53.959 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:53.959 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:53.959 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:53.960 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:53.961 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:53.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:53.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:53.962 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:54.003 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:54.003 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:54.282 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:54.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:54.283 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:54.283 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:52.632067Z" 2020-11-19T03:20:54.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:54.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:54.284 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:54.284 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:54.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:54.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:54.284 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:54.285 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:54.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:54.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:54.285 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:54.285 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:54.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:54.286 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:54.287 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:54.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:54.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:54.288 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:54.331 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:54.331 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:54.602 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:54.616 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:54.617 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:54.639 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... ok 2020-11-19T03:20:54.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:54.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:54.641 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:54.681 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:54.682 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:54.916 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:54.955 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:54.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:54.956 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:54.956 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:52.632067Z" 2020-11-19T03:20:54.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:54.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:54.957 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:54.957 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:54.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:54.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:54.957 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:54.958 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:54.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:54.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:54.958 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:54.958 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:54.959 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:54.960 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:54.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:54.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:54.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:54.962 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:55.005 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:55.005 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:55.269 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.285 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:55.287 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:55.288 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:55.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:55.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:55.289 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:55.340 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:55.340 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:55.620 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:55.621 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:55.621 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:55.621 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:55.621 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:55.621 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:55.622 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:55.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:55.622 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:55.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:55.623 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:55.623 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:55.623 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:55.623 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:55.623 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:55.624 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:55.624 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:55.624 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:55.624 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:55.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:55.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:55.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:55.625 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:55.625 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:55.625 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:55.626 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:55.626 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:55.626 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:55.626 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:55.627 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:55.627 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:55.627 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:55.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:55.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:55.628 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:55.628 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:55.628 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:55.628 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:55.628 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:55.629 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:55.629 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:55.629 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:55.629 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:55.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:55.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:55.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:55.630 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:55.630 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:55.630 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:55.631 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:55.631 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:55.631 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:55.631 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:55.631 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:55.632 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:55.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:55.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:55.632 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:55.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:55.633 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:55.633 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:55.633 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:55.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:55.633 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:55.635 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:55.635 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:55.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:55.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:55.637 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:55.680 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:55.681 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:55.946 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:55.946 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:55.946 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:55.947 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:55.947 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:55.947 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:55.947 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:55.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.948 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:55.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:55.948 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:55.948 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:55.948 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:55.949 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:55.949 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:55.949 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:55.949 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:55.950 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:55.950 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:55.950 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:55.951 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:55.951 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:55.951 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:55.951 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:55.951 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:55.952 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:55.952 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:55.952 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:55.952 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:55.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:55.953 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:55.953 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:55.954 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:55.954 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:55.954 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:55.954 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:55.954 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:55.956 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:55.956 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:55.956 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:55.956 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:55.957 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:55.957 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:55.957 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:55.957 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:55.957 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:55.958 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.958 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.958 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:55.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:55.958 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:55.959 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:55.959 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:55.959 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:55.959 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:55.960 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:55.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:55.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:55.962 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:55.998 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:55.998 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:56.252 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:56.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:56.252 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:56.253 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:52.632067Z" 2020-11-19T03:20:56.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:56.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:56.253 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:56.253 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:56.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:56.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:56.254 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:56.254 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:56.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:56.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:56.255 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:56.255 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:56.255 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:56.256 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:56.256 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:56.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:56.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:56.258 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:56.298 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:56.298 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:56.556 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:57.579 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:57.579 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:57.579 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:57.580 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:57.580 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:57.580 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:57.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:57.581 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:57.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:57.581 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:57.581 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:57.581 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:57.582 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:57.582 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:57.582 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:57.582 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:57.583 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:57.583 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:57.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:57.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:57.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:57.584 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:57.584 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:57.584 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:57.584 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:57.584 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:57.585 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:57.585 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:57.585 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:57.585 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:57.585 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:57.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:57.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:57.586 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:57.586 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:57.587 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:57.587 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:57.587 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:57.587 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:57.587 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:57.588 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:57.588 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:57.588 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:57.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:57.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:57.589 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:57.589 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:57.589 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:57.589 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:57.589 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:57.590 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:57.590 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:57.590 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:57.590 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:57.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:57.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:57.591 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:57.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:57.591 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:57.591 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:57.592 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:57.592 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:57.592 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:57.593 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:57.636 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... ok 2020-11-19T03:20:57.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:57.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:57.638 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:57.684 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:57.684 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:57.927 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:57.967 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:57.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:57.969 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:57.969 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:52.632067Z" 2020-11-19T03:20:57.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:57.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:57.971 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:57.972 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:57.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:57.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:57.974 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:57.975 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:57.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:57.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:57.975 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:57.976 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:57.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:57.976 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:57.977 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:20:57.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:57.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:57.979 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:58.023 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:58.024 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:58.318 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:58.319 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:58.319 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:58.319 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:58.320 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:58.320 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:58.320 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:58.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:58.321 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:58.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:58.322 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:58.322 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:58.322 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:58.322 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:58.323 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:58.323 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:58.323 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:58.323 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:58.324 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:58.324 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:58.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:58.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:58.325 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:58.325 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:58.325 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:58.326 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:58.326 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:58.326 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:58.326 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:58.327 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:58.327 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:58.327 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:58.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:58.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:58.328 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:58.328 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:58.328 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:58.329 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:58.329 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:58.329 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:58.330 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:58.331 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:58.331 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:58.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:58.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:58.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:58.332 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:58.332 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:58.332 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:58.333 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:58.333 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:58.333 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:58.333 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:58.334 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:58.334 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:58.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:58.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:58.335 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:58.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:58.335 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:58.335 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:58.336 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:58.336 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:58.336 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:58.337 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:20:58.338 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:20:58.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:58.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:58.340 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:58.382 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:58.382 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:58.642 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:20:58.642 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:20:58.642 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:20:58.642 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:20:58.643 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:20:58.643 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:58.643 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:58.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:58.644 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:20:58.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:58.644 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:20:58.644 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:58.644 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:58.645 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:58.645 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:58.645 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:20:58.645 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:58.645 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:58.646 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:20:58.646 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:58.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:58.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:58.646 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:20:58.647 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:58.647 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:58.647 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:58.647 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:58.647 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:58.648 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:58.648 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:58.648 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:58.648 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:58.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:58.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:58.649 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:20:58.649 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:58.649 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:58.650 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:58.650 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:58.650 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:58.650 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:58.651 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:58.651 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:58.651 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:58.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:58.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:58.652 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:20:58.652 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:20:58.652 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:20:58.652 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:58.652 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:20:58.653 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:20:58.653 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:20:58.653 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:20:58.653 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:20:58.653 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:58.654 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:58.654 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:20:58.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:58.654 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:20:58.655 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:20:58.655 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:20:58.655 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:58.655 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:20:58.656 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:20:58.657 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:58.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:58.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:58.658 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:58.700 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:58.701 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:59.066 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:59.074 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:59.074 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:59.074 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:59.074 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:59.075 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:59.075 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:59.075 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:59.075 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:59.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:59.076 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:59.076 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:59.076 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:59.077 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:59.077 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:59.077 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:59.077 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:59.077 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:59.078 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:59.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.078 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:59.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.079 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:59.079 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:59.079 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:59.079 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.079 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:59.080 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:59.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:59.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:59.082 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:59.119 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:59.119 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:20:59.382 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:20:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:52.632067Z" 2020-11-19T03:20:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:20:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:20:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:20:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:20:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:20:59.386 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:20:59.387 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:20:59.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:59.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:59.389 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:59.428 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:59.428 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:20:59.690 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:20:59.690 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:20:59.690 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:20:59.691 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:20:59.691 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:20:59.691 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:20:59.691 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:59.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.692 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:20:59.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:59.692 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:20:59.692 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:59.693 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:59.693 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756052.632067_1113.1", 2020-11-19T03:20:59.693 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:59.693 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:20:59.693 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:59.694 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:52.632067Z", 2020-11-19T03:20:59.694 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:20:59.694 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:59.695 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:20:59.695 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:59.695 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:59.695 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:20:59.695 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:59.696 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:59.696 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:59.696 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:20:59.696 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:59.696 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:59.697 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:20:59.697 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:59.697 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:59.698 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:59.698 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:59.698 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:59.698 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:59.699 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:59.699 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:59.699 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:20:59.700 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:20:59.700 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:20:59.700 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:20:59.700 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:20:59.700 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:20:59.701 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:20:59.701 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:20:59.701 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:20:59.701 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:20:59.701 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.704 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.705 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:20:59.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:20:59.705 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:20:59.706 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:20:59.706 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:20:59.706 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:20:59.706 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:20:59.708 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:20:59.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:59.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:20:59.778 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:20:59.816 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:20:59.817 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:00.043 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:00.079 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756059.769065_1135.1", 2020-11-19T03:21:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:20:59.769065Z" 2020-11-19T03:21:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:00.083 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:00.084 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:00.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:00.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:00.086 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:00.127 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:00.128 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:00.383 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:00.383 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:00.384 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:00.384 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:00.384 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:00.384 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:00.384 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:00.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:00.385 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:00.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:00.385 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:00.385 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:00.386 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:00.386 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756059.769065_1135.1", 2020-11-19T03:21:00.386 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:00.386 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:00.386 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:00.387 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:20:59.769065Z", 2020-11-19T03:21:00.387 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:00.387 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:00.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:00.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:00.388 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:00.388 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:00.388 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:00.388 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:00.388 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:00.389 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:00.389 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:00.389 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:00.389 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:00.389 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:00.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:00.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:00.390 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:00.390 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:00.391 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:00.391 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:00.391 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:00.391 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:00.391 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:00.392 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:00.392 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:00.392 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:00.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:00.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:00.393 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:00.393 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:00.394 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:00.395 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:00.395 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:00.396 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:00.396 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:00.397 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:00.399 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:00.400 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:00.402 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:00.402 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:00.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:00.403 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:00.404 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:00.404 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:00.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:00.406 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:00.409 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:00.536 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... ok 2020-11-19T03:21:00.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:00.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:00.538 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:00.578 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:00.578 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:00.819 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:00.856 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:00.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:00.857 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756060.528532_1140.1", 2020-11-19T03:21:00.857 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:21:00.528532Z" 2020-11-19T03:21:00.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:00.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:00.858 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:00.858 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:00.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:00.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:00.858 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:00.859 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:00.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:00.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:00.859 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:00.859 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:00.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:00.860 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:00.861 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:21:00.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:00.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:00.862 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:00.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:00.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:01.173 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:01.174 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:01.174 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:01.174 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:01.175 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:01.175 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:01.175 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:01.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:01.175 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:01.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:01.176 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:01.176 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:01.176 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:01.176 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756060.528532_1140.1", 2020-11-19T03:21:01.177 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:01.177 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:01.177 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:01.177 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:00.528532Z", 2020-11-19T03:21:01.177 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:01.178 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:01.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:01.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:01.178 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:01.178 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:01.179 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:01.179 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:01.179 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:01.179 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:01.179 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:01.180 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:01.180 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:01.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:01.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:01.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:01.181 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:01.181 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:01.181 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:01.182 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:01.182 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:01.182 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:01.182 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:01.182 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:01.183 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:01.183 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:01.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:01.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:01.183 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:01.184 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:01.184 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:01.184 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:01.184 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:01.184 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:01.185 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:01.185 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:01.185 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:01.185 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:01.185 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:01.186 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:01.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:01.186 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:01.186 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:01.186 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:01.187 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:01.187 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:01.188 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:21:01.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:21:01.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:01.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:01.190 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:01.231 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:21:01.517 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:21:01.518 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:21:01.518 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:21:01.518 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:21:01.518 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:01.518 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:21:01.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:01.519 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:21:01.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:01.519 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:21:01.520 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:01.520 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:01.520 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756060.528532_1140.1", 2020-11-19T03:21:01.520 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:01.520 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:21:01.521 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:01.521 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:21:00.528532Z", 2020-11-19T03:21:01.521 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:21:01.522 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:01.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:01.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:01.523 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:21:01.523 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:01.524 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:01.524 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:01.525 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:01.525 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:21:01.525 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:01.536 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:01.540 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:21:01.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:01.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:01.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:01.551 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:21:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:21:01.562 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:01.563 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:21:01.564 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:21:01.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:01.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:01.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:21:01.573 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:01.573 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:02.278 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:02.279 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:02.280 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:21:02.280 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:02.281 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:21:02.282 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:21:02.282 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:02.284 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:02.285 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:21:02.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:02.289 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:21:02.290 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:21:02.294 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:21:02.294 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:21:02.300 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:21:02.300 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:02.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:02.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:02.302 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:02.344 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:02.344 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:02.708 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:02.708 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:02.709 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:02.709 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:02.709 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:02.710 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:02.710 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:02.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:02.710 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:02.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:02.711 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:02.711 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:02.711 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:02.711 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756060.528532_1140.1", 2020-11-19T03:21:02.712 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:02.712 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:02.712 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:02.712 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:00.528532Z", 2020-11-19T03:21:02.712 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:02.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:02.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:02.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:02.713 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:02.714 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:02.714 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:02.714 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:02.714 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:02.714 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:02.715 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:02.715 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:02.715 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:02.715 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:02.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:02.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:02.718 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:02.719 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:02.719 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:02.719 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:02.720 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:02.720 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:02.721 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:02.721 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:02.722 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:02.723 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:02.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:02.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:02.725 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:02.725 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:02.725 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:02.726 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:02.726 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:02.726 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:02.727 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:02.727 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:02.728 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:02.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:02.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:02.729 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:02.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:02.729 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:02.730 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:02.730 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:02.730 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:02.731 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:02.733 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:02.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:02.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:02.735 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:02.779 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:02.780 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:03.033 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:03.075 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756060.528532_1140.1", 2020-11-19T03:21:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:21:00.528532Z" 2020-11-19T03:21:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:03.079 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:03.080 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:03.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:03.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:03.081 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:03.132 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:03.132 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:03.393 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:03.394 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:03.394 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:03.394 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:03.395 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:03.395 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:03.395 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:03.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:03.395 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:03.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:03.396 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:03.396 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:03.396 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:03.396 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756060.528532_1140.1", 2020-11-19T03:21:03.397 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:03.397 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:03.397 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:03.397 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:00.528532Z", 2020-11-19T03:21:03.397 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:03.398 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:03.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:03.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:03.398 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:03.398 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:03.399 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:03.399 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:03.399 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:03.400 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:03.400 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:03.401 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:03.401 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:03.401 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:03.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:03.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:03.402 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:03.402 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:03.402 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:03.402 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:03.403 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:03.403 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:03.403 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:03.403 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:03.403 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:03.404 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:03.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:03.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:03.404 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:03.404 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:03.405 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:03.405 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:03.405 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:03.405 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:03.405 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:03.406 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:03.406 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:03.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:03.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:03.406 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:03.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:03.407 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:03.407 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:03.407 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:03.407 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:03.408 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:03.409 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:03.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:03.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:03.471 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:03.508 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:03.508 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:03.730 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:03.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:03.770 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756063.461055_1150.1", 2020-11-19T03:21:03.771 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:21:03.461055Z" 2020-11-19T03:21:03.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:03.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:03.772 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:03.772 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:03.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:03.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:03.772 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:03.773 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:03.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:03.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:03.773 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:03.774 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:03.774 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:03.774 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:03.775 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:03.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:03.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:03.777 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:03.817 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:03.817 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:04.181 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:04.181 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:04.181 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:04.182 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:04.182 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:04.182 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:04.182 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:04.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.183 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:04.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:04.184 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:04.184 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:04.184 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:04.185 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756063.461055_1150.1", 2020-11-19T03:21:04.185 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:04.185 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:04.185 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:04.186 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:03.461055Z", 2020-11-19T03:21:04.186 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:04.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:04.187 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:04.188 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:04.188 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:04.188 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:04.188 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:04.189 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:04.189 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:04.189 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:04.189 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:04.189 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:04.190 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:04.191 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:04.191 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:04.191 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:04.192 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:04.192 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:04.192 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:04.192 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:04.192 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:04.193 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:04.193 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:04.193 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:04.194 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:04.194 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:04.195 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:04.195 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:04.195 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:04.195 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:04.196 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:04.196 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.196 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.196 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:04.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.197 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:04.197 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:04.197 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:04.197 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.198 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:04.199 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:04.286 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... ok 2020-11-19T03:21:04.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:04.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:04.288 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:04.329 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:04.329 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:04.576 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:04.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:04.616 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756064.278628_1155.1", 2020-11-19T03:21:04.616 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:21:04.278628Z" 2020-11-19T03:21:04.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:04.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:04.617 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:04.617 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:04.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:04.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:04.618 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:04.618 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:04.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:04.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:04.618 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:04.619 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:04.619 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:04.619 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:04.621 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:21:04.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:04.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:04.622 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:04.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:04.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:04.939 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:04.939 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:04.939 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:04.940 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:04.940 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:04.940 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:04.941 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:04.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.941 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:04.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:04.942 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:04.942 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:04.942 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:04.942 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756064.278628_1155.1", 2020-11-19T03:21:04.942 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:04.943 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:04.943 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:04.943 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:04.278628Z", 2020-11-19T03:21:04.943 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:04.944 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:04.944 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:04.944 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:04.945 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:04.945 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:04.945 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:04.945 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:04.946 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:04.946 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:04.946 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:04.946 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:04.947 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:04.947 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:04.947 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:04.947 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:04.948 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:04.948 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:04.948 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:04.948 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:04.948 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:04.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:04.950 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:04.950 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:04.950 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:04.950 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:04.950 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:04.951 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:04.951 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:04.951 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:04.951 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:04.951 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.952 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:04.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:04.952 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:04.952 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:04.953 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:04.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:04.953 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:04.954 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:21:04.955 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:21:04.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:04.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:04.956 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:04.996 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:04.997 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:05.279 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:21:05.280 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:21:05.280 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:21:05.280 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:21:05.280 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:21:05.280 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:05.281 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:21:05.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:05.281 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:21:05.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:05.281 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:21:05.282 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:05.282 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:05.282 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756064.278628_1155.1", 2020-11-19T03:21:05.282 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:05.283 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:21:05.283 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:05.283 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:21:04.278628Z", 2020-11-19T03:21:05.283 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:21:05.283 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:05.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:05.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:05.284 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:21:05.284 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:05.284 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:05.285 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:05.285 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:05.285 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:21:05.285 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:05.285 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:05.286 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:21:05.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:05.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:05.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:05.286 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:21:05.287 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:05.287 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:05.287 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:05.287 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:05.288 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:21:05.288 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:05.288 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:21:05.288 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:21:05.288 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:05.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:05.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:05.289 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:21:05.289 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:05.289 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:05.290 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:05.290 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:05.290 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:21:05.290 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:05.290 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:21:05.291 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:21:05.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:05.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:05.291 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:21:05.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:05.292 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:21:05.292 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:21:05.292 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:21:05.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:05.292 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:21:05.294 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:21:05.294 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:05.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:05.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:05.296 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:05.335 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:05.335 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:05.602 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:05.602 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:05.603 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:05.603 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:05.603 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:05.603 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:05.604 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:05.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:05.604 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:05.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:05.605 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:05.605 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:05.605 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:05.605 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756064.278628_1155.1", 2020-11-19T03:21:05.606 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:05.606 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:05.606 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:05.606 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:04.278628Z", 2020-11-19T03:21:05.607 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:05.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:05.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:05.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:05.608 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:05.608 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:05.608 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:05.608 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:05.609 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:05.609 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:05.609 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:05.609 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:05.609 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:05.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:05.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:05.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:05.610 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:05.611 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:05.611 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:05.611 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:05.612 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:05.612 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:05.612 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:05.612 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:05.612 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:05.613 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:05.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:05.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:05.613 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:05.614 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:05.614 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:05.614 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:05.614 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:05.614 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:05.615 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:05.615 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:05.615 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:05.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:05.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:05.616 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:05.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:05.616 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:05.617 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:05.617 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:05.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:05.617 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:05.619 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:05.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:05.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:05.621 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:05.663 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:05.663 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:05.909 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:05.944 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:05.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:05.945 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756064.278628_1155.1", 2020-11-19T03:21:05.945 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:21:04.278628Z" 2020-11-19T03:21:05.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:05.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:05.946 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:05.946 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:05.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:05.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:05.947 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:05.947 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:05.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:05.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:05.948 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:05.948 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:05.948 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:05.948 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:05.949 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:05.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:05.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:05.951 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:06.005 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:06.006 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:06.279 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:06.280 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:06.280 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:06.280 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:06.280 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:06.281 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:06.281 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:06.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.281 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:06.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:06.282 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:06.282 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:06.282 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:06.282 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756064.278628_1155.1", 2020-11-19T03:21:06.282 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:06.283 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:06.283 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:06.283 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:04.278628Z", 2020-11-19T03:21:06.283 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:06.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:06.284 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:06.284 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:06.284 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:06.285 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:06.285 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:06.285 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:06.285 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:06.287 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:06.287 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:06.287 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:06.287 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:06.287 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:06.288 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:06.288 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:06.288 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:06.288 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:06.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:06.289 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:06.289 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:06.290 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:06.290 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:06.290 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:06.290 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:06.291 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:06.291 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:06.291 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:06.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.292 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:06.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.292 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:06.292 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:06.292 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:06.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.293 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:06.294 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:06.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:06.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:06.359 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:06.396 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:06.396 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:06.637 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:06.675 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756066.350973_1165.1", 2020-11-19T03:21:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:21:06.350973Z" 2020-11-19T03:21:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:06.679 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:06.680 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:06.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:06.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:06.682 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:06.723 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:06.723 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:06.978 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:06.978 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:06.978 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:06.978 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:06.979 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:06.979 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:06.979 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:06.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.980 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:06.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:06.980 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:06.980 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:06.980 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:06.981 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756066.350973_1165.1", 2020-11-19T03:21:06.981 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:06.982 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:06.982 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:06.982 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:06.350973Z", 2020-11-19T03:21:06.982 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:06.982 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:06.983 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:06.983 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:06.984 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:06.984 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:06.984 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:06.984 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:06.984 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:06.985 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:06.985 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:06.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:06.986 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:06.986 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:06.986 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:06.986 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:06.986 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:06.987 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:06.987 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:06.987 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:06.987 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:06.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:06.988 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:06.988 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:06.989 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:06.989 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:06.989 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:06.989 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:06.989 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:06.990 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:06.990 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:06.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.991 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:06.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:06.991 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:06.991 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:06.991 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:06.992 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:06.993 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:06.994 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:07.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:21:07.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:21:07.497 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:21:07.557 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:07.557 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:07.772+0000 7f084eaa10c0 1 waiting for incremental sync to catch up: 2020-11-19T03:21:07.781 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.106.5, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000001.97.5] 2020-11-19T03:21:07.781 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.109.3, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000002.99.5] 2020-11-19T03:21:12.785 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:12.776+0000 7f084eaa10c0 1 waiting for incremental sync to catch up: 2020-11-19T03:21:13.467 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.106.5, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000001.97.5] 2020-11-19T03:21:13.467 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.109.3, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000002.99.5] 2020-11-19T03:21:17.790 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:17.780+0000 7f084eaa10c0 1 bucket sync caught up with source: 2020-11-19T03:21:17.790 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.109.3, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000002.99.5] 2020-11-19T03:21:17.791 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.109.3, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000002.99.5] 2020-11-19T03:21:17.791 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:17.780+0000 7f084eaa10c0 0 bucket checkpoint complete 2020-11-19T03:21:17.965 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2020-11-19T03:21:18.823 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2020-11-19T03:21:18.824 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2020-11-19T03:21:18.825 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2020-11-19T03:21:18.827 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2020-11-19T03:21:18.829 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2020-11-19T03:21:18.829 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2020-11-19T03:21:18.830 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2020-11-19T03:21:18.830 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:21:18.831 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:21:18.831 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-11-19T03:21:18.843 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-19T03:21:18.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:21:18.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:21:18.950 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:21:18.999 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:19.259 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:19.252+0000 7fd9e75260c0 1 waiting for incremental sync to catch up: 2020-11-19T03:21:19.260 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.109.3, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000002.99.5] 2020-11-19T03:21:19.260 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.113.5, , 00000000002.101.5, 00000000002.101.5, 00000000002.118.5, , 00000000002.98.5, 00000000002.74.5, 00000000002.143.5, 00000000004.103.5] 2020-11-19T03:21:24.263 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:24.256+0000 7fd9e75260c0 1 waiting for incremental sync to catch up: 2020-11-19T03:21:24.872 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.109.3, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000002.99.5] 2020-11-19T03:21:24.873 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.113.5, , 00000000002.101.5, 00000000002.101.5, 00000000002.118.5, , 00000000002.98.5, 00000000002.74.5, 00000000002.143.5, 00000000004.103.5] 2020-11-19T03:21:29.268 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:29.260+0000 7fd9e75260c0 1 waiting for incremental sync to catch up: 2020-11-19T03:21:30.206 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.109.3, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000002.99.5] 2020-11-19T03:21:30.206 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.113.5, , 00000000002.101.5, 00000000002.101.5, 00000000002.118.5, , 00000000002.98.5, 00000000002.74.5, 00000000002.143.5, 00000000004.103.5] 2020-11-19T03:21:34.273 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:34.264+0000 7fd9e75260c0 1 waiting for incremental sync to catch up: 2020-11-19T03:21:35.758 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.109.3, , 00000000001.99.5, 00000000001.99.5, 00000000001.116.5, , 00000000001.96.5, 00000000001.72.5, 00000000001.141.5, 00000000002.99.5] 2020-11-19T03:21:35.758 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.113.5, , 00000000002.101.5, 00000000002.101.5, 00000000002.118.5, , 00000000002.98.5, 00000000002.74.5, 00000000002.143.5, 00000000004.103.5] 2020-11-19T03:21:39.277 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:39.268+0000 7fd9e75260c0 1 bucket sync caught up with source: 2020-11-19T03:21:39.277 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000005.113.5, , 00000000002.101.5, 00000000002.101.5, 00000000002.118.5, , 00000000002.98.5, 00000000002.74.5, 00000000002.143.5, 00000000004.103.5] 2020-11-19T03:21:39.278 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.113.5, , 00000000002.101.5, 00000000002.101.5, 00000000002.118.5, , 00000000002.98.5, 00000000002.74.5, 00000000002.143.5, 00000000004.103.5] 2020-11-19T03:21:39.278 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:39.268+0000 7fd9e75260c0 0 bucket checkpoint complete 2020-11-19T03:21:39.522 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... ok 2020-11-19T03:21:39.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:39.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:39.525 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:39.592 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:39.592 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:40.962 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:40.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:40.963 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756099.513276_1269.1", 2020-11-19T03:21:40.964 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:21:39.513276Z" 2020-11-19T03:21:40.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:40.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:40.965 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:40.968 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:40.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:40.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:40.969 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:40.969 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:40.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:40.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:40.970 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:40.971 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:40.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:40.972 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:40.973 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:21:40.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:40.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:40.975 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:41.019 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:41.020 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:41.299 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:41.299 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:41.300 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:41.300 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:41.300 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:41.301 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:41.301 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:41.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:41.301 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:41.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:41.302 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:41.302 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:41.303 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:41.303 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756099.513276_1269.1", 2020-11-19T03:21:41.303 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:41.304 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:41.304 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:41.304 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:39.513276Z", 2020-11-19T03:21:41.305 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:41.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:41.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:41.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:41.306 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:41.306 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:41.306 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:41.307 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:41.307 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:41.307 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:41.308 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:41.308 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:41.309 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:41.309 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:41.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:41.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:41.310 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:41.310 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:41.311 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:41.311 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:41.311 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:41.312 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:41.312 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:41.312 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:41.313 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:41.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:41.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:41.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:41.314 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:41.314 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:41.315 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:41.315 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:41.315 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:41.316 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:41.316 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:41.316 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:41.317 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:41.317 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:41.317 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:41.318 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:41.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:41.318 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:41.319 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:41.319 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:41.319 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:41.319 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:41.320 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:21:41.321 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:21:41.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:41.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:41.323 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:41.366 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:41.367 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:41.662 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:21:41.663 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:21:41.663 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:21:41.663 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:21:41.664 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:21:41.664 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:41.664 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:21:41.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:41.664 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:21:41.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:41.665 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:21:41.665 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:41.665 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:41.666 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756099.513276_1269.1", 2020-11-19T03:21:41.666 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:41.666 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:21:41.666 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:41.666 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:21:39.513276Z", 2020-11-19T03:21:41.667 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:21:41.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:41.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:41.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:41.667 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:21:41.668 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:41.668 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:41.668 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:41.668 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:41.668 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:21:41.669 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:41.669 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:41.669 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:21:41.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:41.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:41.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:41.670 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:21:41.670 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:41.670 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:41.671 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:41.671 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:41.671 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:21:41.671 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:41.671 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:21:41.672 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:21:41.672 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:41.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:41.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:41.672 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:21:41.673 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:21:41.673 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:21:41.673 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:41.673 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:21:41.673 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:21:41.674 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:21:41.674 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:21:41.674 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:21:41.674 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:41.675 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:41.675 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:21:41.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:41.676 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:21:41.677 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:21:41.677 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:21:41.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:41.677 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:21:41.679 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:21:41.679 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:41.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:41.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:41.681 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:41.720 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:41.720 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:42.083 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:42.084 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:42.084 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:42.084 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:42.084 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:42.085 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:42.085 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:42.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.085 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:42.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:42.086 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:42.086 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:42.086 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:42.086 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756099.513276_1269.1", 2020-11-19T03:21:42.087 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:42.087 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:42.087 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:42.087 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:39.513276Z", 2020-11-19T03:21:42.088 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:42.088 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:42.089 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:42.089 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:42.090 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:42.090 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:42.090 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:42.090 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:42.090 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:42.091 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:42.091 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:42.091 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:42.092 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:42.092 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:42.092 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:42.093 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:42.093 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:42.093 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:42.093 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:42.094 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:42.094 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:42.094 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:42.095 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:42.095 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:42.095 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:42.095 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:42.096 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:42.096 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:42.096 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:42.096 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:42.096 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:42.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.097 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:42.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.098 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:42.098 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:42.098 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:42.098 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.098 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:42.100 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:42.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:42.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:42.102 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:42.145 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:42.146 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:42.347 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:42.378 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:42.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:42.379 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756099.513276_1269.1", 2020-11-19T03:21:42.379 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:21:39.513276Z" 2020-11-19T03:21:42.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:42.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:42.380 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:42.380 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:42.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:42.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:42.381 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:42.381 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:42.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:42.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:42.382 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:42.382 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:42.382 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:42.382 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:42.384 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:42.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:42.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:42.386 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:42.427 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:42.428 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:42.802 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:42.802 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:42.803 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:42.803 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:42.803 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:42.803 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:42.804 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:42.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.804 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:42.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:42.804 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:42.805 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:42.805 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:42.805 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756099.513276_1269.1", 2020-11-19T03:21:42.805 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:42.805 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:42.806 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:42.806 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:39.513276Z", 2020-11-19T03:21:42.806 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:42.806 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:42.807 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:42.808 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:42.808 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:42.808 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:42.808 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:42.808 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:42.809 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:42.809 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:42.809 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:42.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:42.810 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:42.810 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:42.810 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:42.811 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:42.811 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:42.811 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:42.811 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:42.811 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:42.812 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:42.812 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:42.812 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:42.813 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:42.813 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:42.813 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:42.813 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:42.813 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:42.814 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:42.814 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:42.814 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:42.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.815 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:42.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:42.815 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:42.815 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:42.815 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:42.816 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:42.816 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:42.817 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:42.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:42.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:42.831 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic list --uid rgw-multisite-test-user --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:21:42.894 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:42.895 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:43.164 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "topics": [ 2020-11-19T03:21:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "topic": { 2020-11-19T03:21:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "user": "", 2020-11-19T03:21:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82_topic", 2020-11-19T03:21:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "dest": { 2020-11-19T03:21:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_name": "", 2020-11-19T03:21:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "oid_prefix": "", 2020-11-19T03:21:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "push_endpoint": "", 2020-11-19T03:21:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "push_endpoint_args": "", 2020-11-19T03:21:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "push_endpoint_topic": "atsuux-82_topic", 2020-11-19T03:21:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "stored_secret": "false", 2020-11-19T03:21:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "persistent": "false" 2020-11-19T03:21:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "arn": "arn:aws:sns:test-zonegroup::atsuux-82_topic", 2020-11-19T03:21:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "opaqueData": "" 2020-11-19T03:21:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "subs": [] 2020-11-19T03:21:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:43.168 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:43.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'atsuux-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:43.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'atsuux-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:43.170 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic get --uid rgw-multisite-test-user --topic atsuux-82_topic --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:21:43.258 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:43.258 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:43.524 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "topic": { 2020-11-19T03:21:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "user": "", 2020-11-19T03:21:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82_topic", 2020-11-19T03:21:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "dest": { 2020-11-19T03:21:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_name": "", 2020-11-19T03:21:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "oid_prefix": "", 2020-11-19T03:21:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "push_endpoint": "", 2020-11-19T03:21:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "push_endpoint_args": "", 2020-11-19T03:21:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "push_endpoint_topic": "atsuux-82_topic", 2020-11-19T03:21:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "stored_secret": "false", 2020-11-19T03:21:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "persistent": "false" 2020-11-19T03:21:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "arn": "arn:aws:sns:test-zonegroup::atsuux-82_topic", 2020-11-19T03:21:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "opaqueData": "" 2020-11-19T03:21:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: "subs": [] 2020-11-19T03:21:43.528 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:43.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:43.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:43.557 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic list --uid rgw-multisite-test-user --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:21:43.600 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:43.601 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:43.895 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:21:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "topics": [ 2020-11-19T03:21:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "topic": { 2020-11-19T03:21:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "user": "", 2020-11-19T03:21:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-82_topic", 2020-11-19T03:21:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "dest": { 2020-11-19T03:21:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_name": "", 2020-11-19T03:21:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "oid_prefix": "", 2020-11-19T03:21:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "push_endpoint": "amqp://127.0.0.1:7001/vhost_1", 2020-11-19T03:21:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "push_endpoint_args": "Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1", 2020-11-19T03:21:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "push_endpoint_topic": "atsuux-82_topic", 2020-11-19T03:21:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "stored_secret": "false", 2020-11-19T03:21:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": "false" 2020-11-19T03:21:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "arn": "arn:aws:sns:test-zonegroup::atsuux-82_topic", 2020-11-19T03:21:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "opaqueData": "" 2020-11-19T03:21:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "subs": [] 2020-11-19T03:21:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:21:43.899 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:21:43.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'atsuux-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:43.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'atsuux-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:43.901 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic get --uid rgw-multisite-test-user --topic atsuux-82_topic --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:21:43.982 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:43.982 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:44.256 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:21:44.256 INFO:teuthology.orchestra.run.smithi086.stdout: "topic": { 2020-11-19T03:21:44.257 INFO:teuthology.orchestra.run.smithi086.stdout: "user": "", 2020-11-19T03:21:44.257 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "atsuux-82_topic", 2020-11-19T03:21:44.257 INFO:teuthology.orchestra.run.smithi086.stdout: "dest": { 2020-11-19T03:21:44.257 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_name": "", 2020-11-19T03:21:44.258 INFO:teuthology.orchestra.run.smithi086.stdout: "oid_prefix": "", 2020-11-19T03:21:44.258 INFO:teuthology.orchestra.run.smithi086.stdout: "push_endpoint": "amqp://127.0.0.1:7001/vhost_1", 2020-11-19T03:21:44.258 INFO:teuthology.orchestra.run.smithi086.stdout: "push_endpoint_args": "Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1", 2020-11-19T03:21:44.258 INFO:teuthology.orchestra.run.smithi086.stdout: "push_endpoint_topic": "atsuux-82_topic", 2020-11-19T03:21:44.258 INFO:teuthology.orchestra.run.smithi086.stdout: "stored_secret": "false", 2020-11-19T03:21:44.259 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": "false" 2020-11-19T03:21:44.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:44.259 INFO:teuthology.orchestra.run.smithi086.stdout: "arn": "arn:aws:sns:test-zonegroup::atsuux-82_topic", 2020-11-19T03:21:44.259 INFO:teuthology.orchestra.run.smithi086.stdout: "opaqueData": "" 2020-11-19T03:21:44.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:44.260 INFO:teuthology.orchestra.run.smithi086.stdout: "subs": [] 2020-11-19T03:21:44.260 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:21:44.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:44.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:44.309 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:44.354 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:44.355 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:21:44.649 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:21:44.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:44.649 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756104.301593_1287.1", 2020-11-19T03:21:44.649 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:21:44.301593Z" 2020-11-19T03:21:44.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:44.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:44.650 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:44.651 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:44.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:44.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:44.651 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:44.652 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:44.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:21:44.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:21:44.652 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:21:44.653 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:21:44.653 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:21:44.653 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:21:44.654 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:21:44.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:44.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:21:44.656 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:21:44.695 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:44.696 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:45.062 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:21:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:21:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:21:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:21:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:21:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:21:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:21:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756104.301593_1287.1", 2020-11-19T03:21:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:21:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:44.301593Z", 2020-11-19T03:21:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:21:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:21:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:21:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:21:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:21:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:21:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:21:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:21:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:21:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:21:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:21:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:21:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:21:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:21:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:21:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:21:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:21:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:21:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:21:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:21:45.075 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:45.076 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:21:45.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:45.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:21:45.123 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription get --uid rgw-multisite-test-user --subscription atsuux-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:21:45.179 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:45.181 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:45.441 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:21:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: "user": "", 2020-11-19T03:21:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82_sub", 2020-11-19T03:21:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: "topic": "atsuux-82_topic", 2020-11-19T03:21:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "dest": { 2020-11-19T03:21:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_name": "pubsub-rgw-multisite-test-user-atsuux-82_topic", 2020-11-19T03:21:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "oid_prefix": "atsuux-82_sub/", 2020-11-19T03:21:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "push_endpoint": "", 2020-11-19T03:21:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "push_endpoint_args": "topic=atsuux-82_topic", 2020-11-19T03:21:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "push_endpoint_topic": "atsuux-82_topic", 2020-11-19T03:21:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "stored_secret": "false", 2020-11-19T03:21:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "persistent": "false" 2020-11-19T03:21:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:21:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "s3_id": "" 2020-11-19T03:21:46.341 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:21:51.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:21:51.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:21:51.801 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-82 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:21:51.853 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:21:51.854 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:21:52.109 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:52.100+0000 7fe9f3ce70c0 1 waiting for incremental sync to catch up: 2020-11-19T03:21:52.109 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.80.5, 00000000001.95.5, 00000000001.112.5, 00000000001.146.5, 00000000001.79.5, 00000000001.116.5, 00000000001.89.5, 00000000001.77.5, 00000000001.106.5, 00000000001.73.5, 00000000002.141.5] 2020-11-19T03:21:52.109 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000011.100.5, 00000000011.115.5, 00000000010.130.5, 00000000010.164.5, 00000000009.95.5, 00000000007.128.5, 00000000010.107.5, 00000000010.95.5, 00000000011.126.5, 00000000011.93.5, 00000000010.157.5] 2020-11-19T03:21:57.113 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:21:57.104+0000 7fe9f3ce70c0 1 waiting for incremental sync to catch up: 2020-11-19T03:21:57.206 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.80.5, 00000000001.95.5, 00000000001.112.5, 00000000001.146.5, 00000000001.79.5, 00000000001.116.5, 00000000001.89.5, 00000000001.77.5, 00000000001.106.5, 00000000001.73.5, 00000000002.141.5] 2020-11-19T03:21:57.206 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000011.100.5, 00000000011.115.5, 00000000010.130.5, 00000000010.164.5, 00000000009.95.5, 00000000007.128.5, 00000000010.107.5, 00000000010.95.5, 00000000011.126.5, 00000000011.93.5, 00000000010.157.5] 2020-11-19T03:22:02.118 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:02.108+0000 7fe9f3ce70c0 1 bucket sync caught up with source: 2020-11-19T03:22:02.603 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000011.100.5, 00000000011.115.5, 00000000010.130.5, 00000000010.164.5, 00000000009.95.5, 00000000007.128.5, 00000000010.107.5, 00000000010.95.5, 00000000011.126.5, 00000000011.93.5, 00000000010.157.5] 2020-11-19T03:22:02.603 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000011.100.5, 00000000011.115.5, 00000000010.130.5, 00000000010.164.5, 00000000009.95.5, 00000000007.128.5, 00000000010.107.5, 00000000010.95.5, 00000000011.126.5, 00000000011.93.5, 00000000010.157.5] 2020-11-19T03:22:02.603 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:02.108+0000 7fe9f3ce70c0 0 bucket checkpoint complete 2020-11-19T03:22:02.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:02.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:02.606 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription atsuux-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:22:02.648 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:02.648 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:02.925 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:22:02.926 INFO:teuthology.orchestra.run.smithi160.stdout: "next_marker": "atsuux-82_sub/1605756119.236401.b2fd77a4", 2020-11-19T03:22:02.926 INFO:teuthology.orchestra.run.smithi160.stdout: "is_truncated": "true", 2020-11-19T03:22:02.927 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2020-11-19T03:22:02.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.927 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756106.603065.dea871ea", 2020-11-19T03:22:02.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.928 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:46.603065Z", 2020-11-19T03:22:02.928 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.929 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.929 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.514089Z" 2020-11-19T03:22:02.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.930 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.930 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.930 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.931 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.931 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.932 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.932 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "4" 2020-11-19T03:22:02.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.933 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.935 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756106.603270.8eab2df8", 2020-11-19T03:22:02.935 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.935 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:46.603270Z", 2020-11-19T03:22:02.936 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.936 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.936 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.412965Z" 2020-11-19T03:22:02.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.937 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.940 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.940 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.941 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.941 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.941 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.942 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "2" 2020-11-19T03:22:02.942 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.942 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.943 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756106.608300.c6b5dd27", 2020-11-19T03:22:02.943 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.943 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:46.608300Z", 2020-11-19T03:22:02.943 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.943 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.944 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.366072Z" 2020-11-19T03:22:02.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.944 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.944 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.944 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.945 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.945 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.945 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.945 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "1" 2020-11-19T03:22:02.946 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.946 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.946 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756106.642763.886303d1", 2020-11-19T03:22:02.947 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.947 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:46.642763Z", 2020-11-19T03:22:02.947 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.947 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.948 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.467720Z" 2020-11-19T03:22:02.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.948 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.948 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.949 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.949 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.949 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.950 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.950 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "3" 2020-11-19T03:22:02.950 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.950 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.951 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756106.675673.ecf8e100", 2020-11-19T03:22:02.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.951 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:46.675673Z", 2020-11-19T03:22:02.951 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.952 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.952 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.346187Z" 2020-11-19T03:22:02.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.952 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.952 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.953 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.953 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.953 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.953 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.954 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "0" 2020-11-19T03:22:02.954 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.954 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.955 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756106.833815.1dbda017", 2020-11-19T03:22:02.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.956 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:46.833815Z", 2020-11-19T03:22:02.956 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.956 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.956 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.560118Z" 2020-11-19T03:22:02.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.957 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.957 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.957 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.957 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.958 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.958 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.958 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "5" 2020-11-19T03:22:02.958 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.959 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.959 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756106.834576.031e27ba", 2020-11-19T03:22:02.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.960 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:46.834576Z", 2020-11-19T03:22:02.960 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.960 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.961 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.739405Z" 2020-11-19T03:22:02.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.961 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.961 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.962 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.962 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.962 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.962 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.963 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "9" 2020-11-19T03:22:02.963 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.963 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.964 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756106.883781.6693a9d9", 2020-11-19T03:22:02.964 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.964 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:46.883781Z", 2020-11-19T03:22:02.964 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.964 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.965 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.606085Z" 2020-11-19T03:22:02.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.965 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.965 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.966 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.966 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.966 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.966 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.967 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "6" 2020-11-19T03:22:02.967 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.967 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.968 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756106.883995.44c25937", 2020-11-19T03:22:02.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.968 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:46.883995Z", 2020-11-19T03:22:02.968 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.969 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.969 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.653015Z" 2020-11-19T03:22:02.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.969 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.969 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.970 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.970 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.970 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.970 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.971 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "7" 2020-11-19T03:22:02.971 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.971 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.972 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756107.042465.4631127b", 2020-11-19T03:22:02.972 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.972 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:47.042465Z", 2020-11-19T03:22:02.972 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.972 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.973 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.785210Z" 2020-11-19T03:22:02.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.973 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.973 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.973 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.974 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.974 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.974 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.974 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "10" 2020-11-19T03:22:02.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.975 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756107.533875.212d7969", 2020-11-19T03:22:02.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.976 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:47.533875Z", 2020-11-19T03:22:02.976 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.976 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.976 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.407643Z" 2020-11-19T03:22:02.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.977 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.977 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.977 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.977 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.978 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.978 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.978 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "24" 2020-11-19T03:22:02.978 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.979 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.980 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756108.435096.38593161", 2020-11-19T03:22:02.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.980 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:48.435096Z", 2020-11-19T03:22:02.980 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.980 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.981 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.143773Z" 2020-11-19T03:22:02.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.981 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.981 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.982 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.982 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.982 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.982 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.983 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "40" 2020-11-19T03:22:02.983 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.983 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.984 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.959999.6830a5dc", 2020-11-19T03:22:02.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.984 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.959999Z", 2020-11-19T03:22:02.984 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.985 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.985 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.959670Z" 2020-11-19T03:22:02.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.985 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.985 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.986 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.986 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.986 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.986 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.987 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "14" 2020-11-19T03:22:02.987 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.987 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.988 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.966125.1e1d0f61", 2020-11-19T03:22:02.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.988 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.966125Z", 2020-11-19T03:22:02.988 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.989 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.989 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.696495Z" 2020-11-19T03:22:02.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.989 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.990 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.990 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.990 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.991 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.991 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.991 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "8" 2020-11-19T03:22:02.991 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.991 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.992 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.967421.5de08b1d", 2020-11-19T03:22:02.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.992 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.967421Z", 2020-11-19T03:22:02.993 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.993 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.993 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.995599Z" 2020-11-19T03:22:02.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.993 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.994 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.994 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.994 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.994 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.995 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:02.995 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "80" 2020-11-19T03:22:02.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:02.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:02.996 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.967760.4faab752", 2020-11-19T03:22:02.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:02.996 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.967760Z", 2020-11-19T03:22:02.997 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:02.997 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:02.997 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.815398Z" 2020-11-19T03:22:02.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.998 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:02.998 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:02.998 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:02.998 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:02.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:02.999 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:02.999 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.000 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "33" 2020-11-19T03:22:03.000 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.000 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.001 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.967879.5f4866c7", 2020-11-19T03:22:03.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.001 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.967879Z", 2020-11-19T03:22:03.001 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.001 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.002 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.850622Z" 2020-11-19T03:22:03.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.002 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.002 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.002 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.003 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.003 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.003 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.003 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "98" 2020-11-19T03:22:03.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.005 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.967973.74f81da6", 2020-11-19T03:22:03.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.005 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.967973Z", 2020-11-19T03:22:03.005 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.006 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.006 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.327640Z" 2020-11-19T03:22:03.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.006 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.007 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.007 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.007 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.007 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.008 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.008 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "66" 2020-11-19T03:22:03.008 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.008 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.009 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.968348.40576420", 2020-11-19T03:22:03.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.009 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.968348Z", 2020-11-19T03:22:03.009 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.010 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.010 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.512614Z" 2020-11-19T03:22:03.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.011 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.011 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.011 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.011 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.012 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "48" 2020-11-19T03:22:03.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.013 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.968591.eb3f51cc", 2020-11-19T03:22:03.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.013 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.968591Z", 2020-11-19T03:22:03.013 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.013 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.014 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.328534Z" 2020-11-19T03:22:03.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.014 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.014 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.014 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.015 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.015 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.015 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.015 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "44" 2020-11-19T03:22:03.016 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.016 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.017 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.968705.6d8063e4", 2020-11-19T03:22:03.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.017 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.968705Z", 2020-11-19T03:22:03.017 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.018 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.018 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.279860Z" 2020-11-19T03:22:03.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.018 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.018 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.019 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.019 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.019 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.020 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.021 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "43" 2020-11-19T03:22:03.021 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.022 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.022 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.970429.9c1aba23", 2020-11-19T03:22:03.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.023 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.970429Z", 2020-11-19T03:22:03.023 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.023 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.024 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.727548Z" 2020-11-19T03:22:03.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.024 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.024 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.024 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.025 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.025 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.025 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.026 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "31" 2020-11-19T03:22:03.026 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.026 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.027 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.971649.1fa87be6", 2020-11-19T03:22:03.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.027 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.971649Z", 2020-11-19T03:22:03.027 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.027 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.028 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.227400Z" 2020-11-19T03:22:03.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.028 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.028 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.028 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.029 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.029 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.029 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.030 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "20" 2020-11-19T03:22:03.030 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.030 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.031 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.972420.d38d628c", 2020-11-19T03:22:03.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.031 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.972420Z", 2020-11-19T03:22:03.031 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.031 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.032 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.563806Z" 2020-11-19T03:22:03.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.032 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.032 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.032 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.033 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.033 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.033 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.033 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "71" 2020-11-19T03:22:03.034 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.034 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.035 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.972742.35e60a38", 2020-11-19T03:22:03.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.035 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.972742Z", 2020-11-19T03:22:03.035 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.035 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.036 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.915804Z" 2020-11-19T03:22:03.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.036 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.036 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.036 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.037 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.037 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.037 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.037 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "13" 2020-11-19T03:22:03.038 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.038 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.038 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.972859.ee539582", 2020-11-19T03:22:03.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.039 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.972859Z", 2020-11-19T03:22:03.039 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.039 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.040 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.267490Z" 2020-11-19T03:22:03.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.040 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.040 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.040 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.041 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.041 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.041 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.041 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "86" 2020-11-19T03:22:03.042 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.042 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.042 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.973230.0c23e561", 2020-11-19T03:22:03.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.043 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.973230Z", 2020-11-19T03:22:03.043 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.043 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.043 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.635766Z" 2020-11-19T03:22:03.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.044 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.044 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.044 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.045 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.045 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.045 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.045 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "29" 2020-11-19T03:22:03.046 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.046 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.046 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.973318.7b82b889", 2020-11-19T03:22:03.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.047 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.973318Z", 2020-11-19T03:22:03.047 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.047 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.047 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.104080Z" 2020-11-19T03:22:03.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.048 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.048 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.048 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.048 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.049 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.049 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.049 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "61" 2020-11-19T03:22:03.050 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.050 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.050 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.973407.6e8b6960", 2020-11-19T03:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.973407Z", 2020-11-19T03:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.464103Z" 2020-11-19T03:22:03.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.052 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.052 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.052 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.052 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.053 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.053 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.053 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "47" 2020-11-19T03:22:03.053 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.054 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.054 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.973497.dac849a1", 2020-11-19T03:22:03.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.055 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.973497Z", 2020-11-19T03:22:03.055 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.055 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.055 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.283581Z" 2020-11-19T03:22:03.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.056 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.056 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.056 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.056 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.057 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.057 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.057 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "65" 2020-11-19T03:22:03.057 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.058 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.058 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.973937.e9b9f2e5", 2020-11-19T03:22:03.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.059 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.973937Z", 2020-11-19T03:22:03.059 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.059 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.059 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.495584Z" 2020-11-19T03:22:03.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.060 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.060 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.060 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.060 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.061 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.061 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.061 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "26" 2020-11-19T03:22:03.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.062 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.976821.4399c651", 2020-11-19T03:22:03.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.062 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.976821Z", 2020-11-19T03:22:03.063 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.063 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.063 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.664539Z" 2020-11-19T03:22:03.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.063 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.064 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.064 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.065 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.065 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.065 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "94" 2020-11-19T03:22:03.065 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.065 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.066 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.982163.5465ad16", 2020-11-19T03:22:03.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.066 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.982163Z", 2020-11-19T03:22:03.067 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.067 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.067 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.744385Z" 2020-11-19T03:22:03.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.067 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.068 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.068 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.068 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.068 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.069 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.069 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "53" 2020-11-19T03:22:03.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.070 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.982401.2a01c97a", 2020-11-19T03:22:03.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.070 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.982401Z", 2020-11-19T03:22:03.071 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.071 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.071 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:51.391946Z" 2020-11-19T03:22:03.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.071 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.072 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.072 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.072 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.072 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.073 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.073 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "109" 2020-11-19T03:22:03.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.074 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.989493.5e66dac0", 2020-11-19T03:22:03.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.074 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.989493Z", 2020-11-19T03:22:03.075 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.075 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.075 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.147876Z" 2020-11-19T03:22:03.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.075 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.076 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.076 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.076 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.076 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.077 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.077 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "62" 2020-11-19T03:22:03.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.078 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.989697.79a0e172", 2020-11-19T03:22:03.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.078 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.989697Z", 2020-11-19T03:22:03.078 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.079 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.079 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.415511Z" 2020-11-19T03:22:03.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.079 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.080 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.080 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.080 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.080 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.081 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.081 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "68" 2020-11-19T03:22:03.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.082 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.989911.05891e0e", 2020-11-19T03:22:03.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.082 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.989911Z", 2020-11-19T03:22:03.083 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.083 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.083 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.943491Z" 2020-11-19T03:22:03.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.083 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.084 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.084 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.084 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.084 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.085 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.085 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "100" 2020-11-19T03:22:03.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.086 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.995256.7ebc71e4", 2020-11-19T03:22:03.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.086 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.995256Z", 2020-11-19T03:22:03.086 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.087 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.087 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.859671Z" 2020-11-19T03:22:03.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.087 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.087 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.088 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.088 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.088 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.088 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.089 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "34" 2020-11-19T03:22:03.089 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.089 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.090 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.995632.9c8236c1", 2020-11-19T03:22:03.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.090 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.995632Z", 2020-11-19T03:22:03.090 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.091 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.091 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.136283Z" 2020-11-19T03:22:03.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.091 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.091 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.092 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.092 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.092 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.092 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.093 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "83" 2020-11-19T03:22:03.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.094 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.995705.146b59cd", 2020-11-19T03:22:03.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.094 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.995705Z", 2020-11-19T03:22:03.094 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.094 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.095 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.091189Z" 2020-11-19T03:22:03.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.095 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.095 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.095 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.096 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.096 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.096 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.097 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "17" 2020-11-19T03:22:03.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.098 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.995878.857c2c09", 2020-11-19T03:22:03.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.098 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.995878Z", 2020-11-19T03:22:03.098 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.098 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.099 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.003194Z" 2020-11-19T03:22:03.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.099 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.099 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.099 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.100 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.100 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.100 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.100 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "15" 2020-11-19T03:22:03.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.102 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.998710.5883a70a", 2020-11-19T03:22:03.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.102 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.998710Z", 2020-11-19T03:22:03.102 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.102 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.103 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.827745Z" 2020-11-19T03:22:03.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.103 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.103 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.103 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.104 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.104 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.104 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "11" 2020-11-19T03:22:03.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.105 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.999200.93cef7e1", 2020-11-19T03:22:03.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.106 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.999200Z", 2020-11-19T03:22:03.106 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.106 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.106 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.700917Z" 2020-11-19T03:22:03.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.107 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.107 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.107 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.107 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.108 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.108 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.108 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "52" 2020-11-19T03:22:03.108 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.109 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.109 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756118.999870.de9e8eff", 2020-11-19T03:22:03.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.110 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:58.999870Z", 2020-11-19T03:22:03.110 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.110 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.110 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.903588Z" 2020-11-19T03:22:03.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.111 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.111 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.111 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.111 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.112 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.112 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.112 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "35" 2020-11-19T03:22:03.112 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.113 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.113 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.028222.983227e4", 2020-11-19T03:22:03.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.114 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.028222Z", 2020-11-19T03:22:03.114 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.114 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.114 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.516376Z" 2020-11-19T03:22:03.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.115 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.115 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.115 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.115 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.116 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.116 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.116 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "70" 2020-11-19T03:22:03.116 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.116 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.117 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.031939.1169cd68", 2020-11-19T03:22:03.117 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.117 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.031939Z", 2020-11-19T03:22:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.179440Z" 2020-11-19T03:22:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.119 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.119 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.119 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "84" 2020-11-19T03:22:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.032109.539fdb98", 2020-11-19T03:22:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.032109Z", 2020-11-19T03:22:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.043448Z" 2020-11-19T03:22:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "81" 2020-11-19T03:22:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.032880.937bb997", 2020-11-19T03:22:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.032880Z", 2020-11-19T03:22:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.126 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.126 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.371520Z" 2020-11-19T03:22:03.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.126 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.126 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.127 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.127 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.127 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.128 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "67" 2020-11-19T03:22:03.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.040797.1cb80931", 2020-11-19T03:22:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.040797Z", 2020-11-19T03:22:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.424859Z" 2020-11-19T03:22:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.132 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "89" 2020-11-19T03:22:03.132 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.132 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.133 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.040972.4e018240", 2020-11-19T03:22:03.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.133 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.040972Z", 2020-11-19T03:22:03.133 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.133 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.134 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.372157Z" 2020-11-19T03:22:03.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.134 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.134 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.134 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.135 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.135 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.135 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.135 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "45" 2020-11-19T03:22:03.136 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.136 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.136 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.045572.57b9fbda", 2020-11-19T03:22:03.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.137 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.045572Z", 2020-11-19T03:22:03.137 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.137 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.137 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.557687Z" 2020-11-19T03:22:03.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.138 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.138 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.138 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.139 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.139 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.139 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.139 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "49" 2020-11-19T03:22:03.140 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.140 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.140 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.045722.296536ca", 2020-11-19T03:22:03.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.141 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.045722Z", 2020-11-19T03:22:03.141 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.141 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.141 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.223554Z" 2020-11-19T03:22:03.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.142 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.142 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.142 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.142 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.143 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.143 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.143 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "85" 2020-11-19T03:22:03.143 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.144 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.144 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.045779.7c7ab08a", 2020-11-19T03:22:03.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.145 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.045779Z", 2020-11-19T03:22:03.145 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.145 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.145 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.679609Z" 2020-11-19T03:22:03.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.146 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.146 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.146 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.146 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.147 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.147 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.147 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "30" 2020-11-19T03:22:03.147 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.148 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.148 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.047769.580eec3a", 2020-11-19T03:22:03.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.149 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.047769Z", 2020-11-19T03:22:03.149 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.149 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.149 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.897526Z" 2020-11-19T03:22:03.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.150 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.150 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.150 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.150 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.151 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.151 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.151 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "99" 2020-11-19T03:22:03.151 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.151 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.152 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.048071.84b5ecf3", 2020-11-19T03:22:03.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.152 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.048071Z", 2020-11-19T03:22:03.153 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.153 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.153 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:46.871382Z" 2020-11-19T03:22:03.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.153 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.154 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.154 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.154 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.155 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.155 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.155 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "12" 2020-11-19T03:22:03.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.156 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.135925.58e390cf", 2020-11-19T03:22:03.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.156 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.135925Z", 2020-11-19T03:22:03.157 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.157 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.157 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.047073Z" 2020-11-19T03:22:03.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.157 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.158 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.158 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.158 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.158 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.159 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.159 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "16" 2020-11-19T03:22:03.159 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.159 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.160 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.146325.84ad54fc", 2020-11-19T03:22:03.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.160 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.146325Z", 2020-11-19T03:22:03.160 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.161 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.161 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.831947Z" 2020-11-19T03:22:03.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.161 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.161 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.162 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.162 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.162 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.163 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.163 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "55" 2020-11-19T03:22:03.163 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.163 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.164 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.146502.b00e6700", 2020-11-19T03:22:03.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.164 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.146502Z", 2020-11-19T03:22:03.164 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.165 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.165 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.191682Z" 2020-11-19T03:22:03.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.165 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.165 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.166 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.166 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.166 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.166 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.167 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "63" 2020-11-19T03:22:03.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.168 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.150603.42430cb3", 2020-11-19T03:22:03.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.168 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.150603Z", 2020-11-19T03:22:03.168 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.169 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.169 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.902772Z" 2020-11-19T03:22:03.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.169 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.169 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.170 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.170 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.170 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.170 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.171 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "78" 2020-11-19T03:22:03.171 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.171 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.172 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.155909.4ae5842a", 2020-11-19T03:22:03.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.172 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.155909Z", 2020-11-19T03:22:03.172 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.172 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.173 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.516320Z" 2020-11-19T03:22:03.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.173 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.173 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.173 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.174 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.174 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.174 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.175 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "91" 2020-11-19T03:22:03.175 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.175 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.176 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.156086.cb6b554a", 2020-11-19T03:22:03.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.176 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.156086Z", 2020-11-19T03:22:03.176 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.176 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.177 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.275584Z" 2020-11-19T03:22:03.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.177 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.177 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.177 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.178 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.178 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.178 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.178 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "21" 2020-11-19T03:22:03.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.179 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.156277.0c4d8a48", 2020-11-19T03:22:03.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.180 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.156277Z", 2020-11-19T03:22:03.180 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.180 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.180 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.587766Z" 2020-11-19T03:22:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.182 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.182 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.182 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "28" 2020-11-19T03:22:03.182 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.183 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.183 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.156365.d006e122", 2020-11-19T03:22:03.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.184 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.156365Z", 2020-11-19T03:22:03.184 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.184 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.184 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:51.138879Z" 2020-11-19T03:22:03.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.185 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.185 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.185 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.185 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.186 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.186 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.186 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "104" 2020-11-19T03:22:03.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.187 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.187 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.156604.1f031b14", 2020-11-19T03:22:03.187 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.188 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.156604Z", 2020-11-19T03:22:03.188 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.188 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.188 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.759336Z" 2020-11-19T03:22:03.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.189 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.189 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.189 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.189 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.190 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.190 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.190 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "96" 2020-11-19T03:22:03.190 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.190 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.191 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.156691.16a2d39a", 2020-11-19T03:22:03.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.191 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.156691Z", 2020-11-19T03:22:03.192 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.192 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.192 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:51.235865Z" 2020-11-19T03:22:03.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.192 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.193 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.193 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.193 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.194 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.194 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.194 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "106" 2020-11-19T03:22:03.194 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.194 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.195 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.156803.a21a844e", 2020-11-19T03:22:03.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.195 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.156803Z", 2020-11-19T03:22:03.196 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.196 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.196 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.564858Z" 2020-11-19T03:22:03.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.196 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.197 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.197 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.197 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.197 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.198 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.198 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "92" 2020-11-19T03:22:03.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.199 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.156857.af825e2e", 2020-11-19T03:22:03.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.199 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.156857Z", 2020-11-19T03:22:03.199 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.200 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.200 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.856954Z" 2020-11-19T03:22:03.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.200 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.200 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.201 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.201 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.201 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.201 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.202 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "77" 2020-11-19T03:22:03.202 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.202 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.203 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.157120.c7c5c650", 2020-11-19T03:22:03.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.203 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.157120Z", 2020-11-19T03:22:03.203 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.204 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.204 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.990613Z" 2020-11-19T03:22:03.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.204 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.204 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.205 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.205 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.205 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.205 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.206 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "101" 2020-11-19T03:22:03.206 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.206 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.206 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.207 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.159809.f5bfa2fa", 2020-11-19T03:22:03.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.207 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.159809Z", 2020-11-19T03:22:03.207 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.207 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.208 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.371417Z" 2020-11-19T03:22:03.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.208 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.208 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.208 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.209 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.209 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.209 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.209 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "88" 2020-11-19T03:22:03.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.211 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.160063.18b0a280", 2020-11-19T03:22:03.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.211 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.160063Z", 2020-11-19T03:22:03.211 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.211 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.212 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.923755Z" 2020-11-19T03:22:03.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.212 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.212 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.212 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.213 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.213 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.213 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.213 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "57" 2020-11-19T03:22:03.214 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.214 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.214 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.214 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.160226.26baadec", 2020-11-19T03:22:03.215 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.215 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.160226Z", 2020-11-19T03:22:03.215 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.215 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.215 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:51.037974Z" 2020-11-19T03:22:03.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.216 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.216 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.216 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.216 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.217 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.217 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.217 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "102" 2020-11-19T03:22:03.217 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.218 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.218 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.160889.a93f91bb", 2020-11-19T03:22:03.218 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.219 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.160889Z", 2020-11-19T03:22:03.219 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.219 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.219 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.608514Z" 2020-11-19T03:22:03.219 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.220 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.220 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.220 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.220 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.221 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.221 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.221 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "50" 2020-11-19T03:22:03.221 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.221 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.222 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.222 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.168165.24fdfb02", 2020-11-19T03:22:03.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.223 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.168165Z", 2020-11-19T03:22:03.223 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.223 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.223 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.095938Z" 2020-11-19T03:22:03.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.224 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.224 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.224 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.224 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.225 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.225 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.225 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "39" 2020-11-19T03:22:03.225 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.225 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.226 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.168443.c8c331c8", 2020-11-19T03:22:03.226 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.226 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.168443Z", 2020-11-19T03:22:03.227 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.227 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.227 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.772084Z" 2020-11-19T03:22:03.227 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.227 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.228 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.228 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.228 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.228 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.229 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.229 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "32" 2020-11-19T03:22:03.229 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.229 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.229 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.230 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.230 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.168932.2094b5bf", 2020-11-19T03:22:03.230 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.230 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.168932Z", 2020-11-19T03:22:03.230 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.231 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.231 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.667760Z" 2020-11-19T03:22:03.231 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.231 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.232 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.232 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.232 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.232 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.233 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.233 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "73" 2020-11-19T03:22:03.233 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.233 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.234 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.234 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.170379.4b20a5cd", 2020-11-19T03:22:03.234 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.234 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.170379Z", 2020-11-19T03:22:03.234 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.235 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.235 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.135283Z" 2020-11-19T03:22:03.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.235 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.235 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.236 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.236 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.236 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.236 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.236 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.237 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "18" 2020-11-19T03:22:03.237 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.237 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.237 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.237 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.238 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.170447.5a93ca7b", 2020-11-19T03:22:03.238 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.238 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.170447Z", 2020-11-19T03:22:03.238 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.238 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.239 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.011360Z" 2020-11-19T03:22:03.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.239 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.239 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.239 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.240 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.240 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.240 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.240 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "59" 2020-11-19T03:22:03.241 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.241 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.241 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.241 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.242 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.170506.c4069700", 2020-11-19T03:22:03.242 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.242 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.170506Z", 2020-11-19T03:22:03.242 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.242 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.243 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.713521Z" 2020-11-19T03:22:03.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.243 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.243 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.243 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.244 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.244 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.244 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.244 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "95" 2020-11-19T03:22:03.245 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.245 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.245 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.170563.b858e5e0", 2020-11-19T03:22:03.246 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.246 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.170563Z", 2020-11-19T03:22:03.246 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.246 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.246 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.951885Z" 2020-11-19T03:22:03.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.247 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.247 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.247 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.247 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.248 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.248 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.248 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "36" 2020-11-19T03:22:03.248 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.249 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.170612.0f834e6c", 2020-11-19T03:22:03.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.250 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.170612Z", 2020-11-19T03:22:03.250 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.250 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.250 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.462690Z" 2020-11-19T03:22:03.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.251 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.251 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.251 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.251 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.252 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.252 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.252 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "69" 2020-11-19T03:22:03.252 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.253 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.174965.fcbc5f66", 2020-11-19T03:22:03.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.254 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.174965Z", 2020-11-19T03:22:03.254 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.254 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.254 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.231836Z" 2020-11-19T03:22:03.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.255 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.255 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.255 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.255 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.256 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.256 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.256 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "42" 2020-11-19T03:22:03.256 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.256 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.257 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.200632.28ad2f21", 2020-11-19T03:22:03.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.258 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.200632Z", 2020-11-19T03:22:03.258 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.258 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.258 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.789191Z" 2020-11-19T03:22:03.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.259 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.259 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.259 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.259 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.260 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.260 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.260 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "54" 2020-11-19T03:22:03.260 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.260 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.261 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.217518.f8236cdf", 2020-11-19T03:22:03.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.261 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.217518Z", 2020-11-19T03:22:03.262 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.262 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.262 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.653106Z" 2020-11-19T03:22:03.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.262 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.263 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.263 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.263 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.263 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.264 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.264 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "51" 2020-11-19T03:22:03.264 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.264 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.265 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.222632.d9082f0e", 2020-11-19T03:22:03.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.265 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.222632Z", 2020-11-19T03:22:03.265 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.266 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.266 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.609658Z" 2020-11-19T03:22:03.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.266 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.266 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.267 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.267 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.267 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.268 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.268 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "93" 2020-11-19T03:22:03.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.269 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.226439.54b7a329", 2020-11-19T03:22:03.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.269 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.226439Z", 2020-11-19T03:22:03.269 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.270 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.270 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.763750Z" 2020-11-19T03:22:03.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.270 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.270 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.271 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.271 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.271 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.271 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.272 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "75" 2020-11-19T03:22:03.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.273 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.226793.1cebc046", 2020-11-19T03:22:03.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.273 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.226793Z", 2020-11-19T03:22:03.273 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.273 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.274 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.543449Z" 2020-11-19T03:22:03.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.274 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.274 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.274 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.275 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.275 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.275 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.275 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "27" 2020-11-19T03:22:03.276 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.276 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.276 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.226868.1687c0df", 2020-11-19T03:22:03.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.277 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.226868Z", 2020-11-19T03:22:03.277 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.277 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.278 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.060191Z" 2020-11-19T03:22:03.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.278 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.278 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.278 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.279 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.279 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.279 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.279 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "60" 2020-11-19T03:22:03.280 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.280 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.280 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.227112.4eafa212", 2020-11-19T03:22:03.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.281 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.227112Z", 2020-11-19T03:22:03.281 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.281 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.281 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.183377Z" 2020-11-19T03:22:03.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.282 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.282 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.282 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.282 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.283 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.283 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.283 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "19" 2020-11-19T03:22:03.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.284 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.284 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.227155.04ad2eab", 2020-11-19T03:22:03.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.285 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.227155Z", 2020-11-19T03:22:03.285 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.285 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.285 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:51.333833Z" 2020-11-19T03:22:03.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.286 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.286 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.286 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.286 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.287 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.287 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.287 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "108" 2020-11-19T03:22:03.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.288 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.227213.485603ec", 2020-11-19T03:22:03.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.288 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.227213Z", 2020-11-19T03:22:03.289 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.289 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.289 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.235865Z" 2020-11-19T03:22:03.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.290 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.290 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.290 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.290 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.291 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.291 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.291 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "64" 2020-11-19T03:22:03.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.292 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.227393.e14febd1", 2020-11-19T03:22:03.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.292 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.227393Z", 2020-11-19T03:22:03.293 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.293 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.293 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.967426Z" 2020-11-19T03:22:03.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.293 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.294 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.294 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.294 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.294 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.295 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.295 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "58" 2020-11-19T03:22:03.295 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.295 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.296 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.227601.b75f9046", 2020-11-19T03:22:03.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.227601Z", 2020-11-19T03:22:03.296 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.297 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.297 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:51.186863Z" 2020-11-19T03:22:03.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.297 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.297 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.298 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.298 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.298 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.298 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.299 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "105" 2020-11-19T03:22:03.299 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.299 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.300 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.232624.03893932", 2020-11-19T03:22:03.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.300 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.232624Z", 2020-11-19T03:22:03.300 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.300 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.301 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:51.085577Z" 2020-11-19T03:22:03.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.301 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.301 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.302 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.302 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.302 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.302 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.303 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "103" 2020-11-19T03:22:03.303 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.303 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.304 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.235177.b09452d9", 2020-11-19T03:22:03.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.304 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.235177Z", 2020-11-19T03:22:03.304 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.304 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.305 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.880323Z" 2020-11-19T03:22:03.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.305 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.305 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.305 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.306 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.306 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.306 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.306 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "56" 2020-11-19T03:22:03.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.307 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.235644.9b5de61e", 2020-11-19T03:22:03.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.308 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.235644Z", 2020-11-19T03:22:03.308 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.308 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.309 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.001483Z" 2020-11-19T03:22:03.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.309 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.309 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.309 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.310 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.310 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.310 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.310 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "37" 2020-11-19T03:22:03.311 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.311 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.311 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.235703.55d00280", 2020-11-19T03:22:03.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.312 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.235703Z", 2020-11-19T03:22:03.312 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.312 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.312 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:51.283369Z" 2020-11-19T03:22:03.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.313 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.313 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.313 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.313 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.314 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.314 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.314 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "107" 2020-11-19T03:22:03.314 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.315 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.315 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.236025.e9700cbd", 2020-11-19T03:22:03.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.316 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.236025Z", 2020-11-19T03:22:03.316 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.316 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.316 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.187478Z" 2020-11-19T03:22:03.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.317 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.317 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.317 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.317 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.318 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.318 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.318 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "41" 2020-11-19T03:22:03.318 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.318 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.319 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.236092.8c399ff5", 2020-11-19T03:22:03.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.319 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.236092Z", 2020-11-19T03:22:03.320 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.320 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.320 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.951879Z" 2020-11-19T03:22:03.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.320 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "79" 2020-11-19T03:22:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.323 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.236155.0ea53d1f", 2020-11-19T03:22:03.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.323 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.236155Z", 2020-11-19T03:22:03.324 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.324 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.324 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.087510Z" 2020-11-19T03:22:03.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.324 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.325 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.325 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.325 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.325 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.326 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.326 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "82" 2020-11-19T03:22:03.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.327 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.236401.b2fd77a4", 2020-11-19T03:22:03.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.328 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.236401Z", 2020-11-19T03:22:03.328 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.329 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.329 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.804969Z" 2020-11-19T03:22:03.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.330 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.330 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.330 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.331 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.331 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "97" 2020-11-19T03:22:03.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.332 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:22:03.332 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:22:03.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--marker', 'atsuux-82_sub/1605756119.236401.b2fd77a4', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:03.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--marker', 'atsuux-82_sub/1605756119.236401.b2fd77a4', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:03.334 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription atsuux-82_sub --marker atsuux-82_sub/1605756119.236401.b2fd77a4 --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:22:03.413 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:03.413 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:03.697 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:22:03.698 INFO:teuthology.orchestra.run.smithi160.stdout: "next_marker": "", 2020-11-19T03:22:03.698 INFO:teuthology.orchestra.run.smithi160.stdout: "is_truncated": "false", 2020-11-19T03:22:03.698 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2020-11-19T03:22:03.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.698 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.250007.caf84e23", 2020-11-19T03:22:03.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.699 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.250007Z", 2020-11-19T03:22:03.699 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.699 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.699 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.451645Z" 2020-11-19T03:22:03.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.700 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.700 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.700 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.701 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.701 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.701 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.701 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "25" 2020-11-19T03:22:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.702 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.250692.0501969b", 2020-11-19T03:22:03.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.703 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.250692Z", 2020-11-19T03:22:03.703 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.703 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.703 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.047750Z" 2020-11-19T03:22:03.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.704 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.704 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.704 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.704 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "38" 2020-11-19T03:22:03.705 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.706 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.706 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.250759.45516a27", 2020-11-19T03:22:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.250759Z", 2020-11-19T03:22:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.707 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.613807Z" 2020-11-19T03:22:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.708 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "72" 2020-11-19T03:22:03.709 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.250823.ecf60719", 2020-11-19T03:22:03.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.250823Z", 2020-11-19T03:22:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.810333Z" 2020-11-19T03:22:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "76" 2020-11-19T03:22:03.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.714 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.714 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.251137.37e06ed7", 2020-11-19T03:22:03.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.715 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.251137Z", 2020-11-19T03:22:03.715 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.715 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.715 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.319521Z" 2020-11-19T03:22:03.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.716 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.716 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.716 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.716 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.717 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.717 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.717 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "22" 2020-11-19T03:22:03.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.718 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.718 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.251303.f7e3608d", 2020-11-19T03:22:03.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.719 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.251303Z", 2020-11-19T03:22:03.719 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.719 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.719 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:49.716157Z" 2020-11-19T03:22:03.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.720 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.720 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.720 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.720 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.721 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.721 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.721 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "74" 2020-11-19T03:22:03.721 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.721 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.722 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.260249.cd54b0a2", 2020-11-19T03:22:03.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.722 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.260249Z", 2020-11-19T03:22:03.723 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.723 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.723 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.469167Z" 2020-11-19T03:22:03.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.723 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.784 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.784 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.785 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.785 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.785 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.786 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "90" 2020-11-19T03:22:03.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.787 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.268244.61edeb1d", 2020-11-19T03:22:03.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.268244Z", 2020-11-19T03:22:03.787 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.788 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.788 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:47.363351Z" 2020-11-19T03:22:03.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.788 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.788 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.789 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.789 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.789 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.789 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.790 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "23" 2020-11-19T03:22:03.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.792 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.280062.5b80e1d0", 2020-11-19T03:22:03.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.792 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.280062Z", 2020-11-19T03:22:03.792 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.793 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.793 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:50.312806Z" 2020-11-19T03:22:03.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.793 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.793 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.794 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.794 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.794 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.794 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.795 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "87" 2020-11-19T03:22:03.795 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.795 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:03.796 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1605756119.291828.a0d15b0b", 2020-11-19T03:22:03.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "OBJECT_CREATE", 2020-11-19T03:22:03.796 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:21:59.291828Z", 2020-11-19T03:22:03.796 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:03.796 INFO:teuthology.orchestra.run.smithi160.stdout: "attrs": { 2020-11-19T03:22:03.797 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2020-11-19T03:21:48.417205Z" 2020-11-19T03:22:03.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.797 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket": { 2020-11-19T03:22:03.797 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_id": "d71482d6-15fd-4715-9992-a25958dcb538.5486.30", 2020-11-19T03:22:03.797 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "atsuux-82", 2020-11-19T03:22:03.798 INFO:teuthology.orchestra.run.smithi160.stdout: "tenant": "" 2020-11-19T03:22:03.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:03.798 INFO:teuthology.orchestra.run.smithi160.stdout: "key": { 2020-11-19T03:22:03.798 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2020-11-19T03:22:03.799 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "46" 2020-11-19T03:22:03.799 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.799 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.799 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:03.799 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:22:03.800 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:22:03.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'ack', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--event-id', '1605756119.250007.caf84e23', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:03.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'ack', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--event-id', '1605756119.250007.caf84e23', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:03.802 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription ack --uid rgw-multisite-test-user --subscription atsuux-82_sub --event-id 1605756119.250007.caf84e23 --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:22:03.886 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:03.886 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:04.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'rm', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:04.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'rm', '--uid', 'rgw-multisite-test-user', '--subscription', 'atsuux-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:04.262 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription rm --uid rgw-multisite-test-user --subscription atsuux-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:22:04.340 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:04.340 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:04.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'atsuux-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:04.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'atsuux-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:04.713 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic rm --uid rgw-multisite-test-user --topic atsuux-82_topic --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:22:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:05.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'atsuux-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:05.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'atsuux-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:22:05.166 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic rm --uid rgw-multisite-test-user --topic atsuux-82_topic --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:22:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:06.596 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... ok 2020-11-19T03:22:06.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:06.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:06.598 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:06.629 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:06.630 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:06.852 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:22:06.886 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:22:06.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:06.887 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756126.589058_1295.1", 2020-11-19T03:22:06.887 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:22:06.589058Z" 2020-11-19T03:22:06.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:06.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:06.888 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:06.888 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:06.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:06.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:06.889 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:06.889 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:06.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:06.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:06.890 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:06.890 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:06.890 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:06.890 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:22:06.891 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:22:06.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:06.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:06.893 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:06.938 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:06.938 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:07.193 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:22:08.578 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:22:08.578 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:08.578 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:22:08.578 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:22:08.579 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:22:08.579 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:08.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:08.579 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:22:08.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:08.580 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:22:08.580 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:08.580 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:08.580 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756126.589058_1295.1", 2020-11-19T03:22:08.581 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:08.581 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:22:08.581 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:08.581 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:22:06.589058Z", 2020-11-19T03:22:08.581 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:08.582 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:08.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:08.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:08.582 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:22:08.582 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:08.583 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:08.583 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:22:08.583 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:08.583 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:08.583 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:08.584 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:22:08.584 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:08.584 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:08.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:08.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:08.585 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:22:08.585 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:08.585 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:08.585 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:08.585 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:08.586 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:08.586 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:08.586 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:08.586 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:08.586 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:08.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:08.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:08.587 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:22:08.587 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:08.587 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:08.588 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:08.588 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:08.588 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:08.588 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:08.589 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:08.589 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:08.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:08.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:08.589 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:22:08.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:08.590 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:22:08.590 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:22:08.590 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:22:08.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:08.591 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:22:08.592 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:22:08.592 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:22:08.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:08.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:08.594 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:08.635 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:08.636 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:08.917 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:22:08.917 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:22:08.917 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:22:08.917 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:22:08.918 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:22:08.918 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:22:08.918 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:22:08.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:22:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:22:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:22:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:22:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756126.589058_1295.1", 2020-11-19T03:22:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:22:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:22:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:22:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:22:06.589058Z", 2020-11-19T03:22:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:22:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:22:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:22:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:22:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:22:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:22:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:22:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:22:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:22:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:22:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:22:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:22:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:22:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:22:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:22:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:22:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:22:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:22:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:22:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:22:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:22:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:22:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:22:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:22:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:22:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:22:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:22:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:22:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:22:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:22:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:08.930 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:22:08.931 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:22:08.932 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:22:08.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:08.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:08.933 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:08.974 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:08.975 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:09.346 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:22:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:22:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:22:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:22:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:22:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:22:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:09.349 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:22:09.349 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:09.349 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756126.589058_1295.1", 2020-11-19T03:22:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:22:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:22:06.589058Z", 2020-11-19T03:22:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:22:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:09.353 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:22:09.353 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:09.353 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:09.353 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:22:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:22:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:22:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:22:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:22:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:22:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:22:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:09.362 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:22:09.363 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:22:09.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:09.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:09.364 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:09.403 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:09.404 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:09.637 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:22:09.674 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:22:09.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:09.674 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756126.589058_1295.1", 2020-11-19T03:22:09.675 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:22:06.589058Z" 2020-11-19T03:22:09.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:09.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:09.676 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:09.676 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:09.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:09.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:09.677 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:09.677 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:09.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:09.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:09.678 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:09.678 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:09.679 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:09.679 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:22:09.680 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:22:09.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:09.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:09.682 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:09.725 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:09.725 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:09.990 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:22:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:22:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:22:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:22:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:22:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:22:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:22:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756126.589058_1295.1", 2020-11-19T03:22:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:22:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:22:06.589058Z", 2020-11-19T03:22:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:22:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:22:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:22:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:22:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:22:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:22:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:22:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:22:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:22:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:10.006 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:22:10.007 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:22:10.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:10.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:10.535 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:22:10.588 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:10.588 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:10.818 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:10.811+0000 7fee871410c0 1 bucket sync caught up with source: 2020-11-19T03:22:10.818 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.106.5, , 00000000001.181.5, 00000000001.121.5, 00000000001.187.5, , 00000000001.76.5, 00000000001.88.5, 00000000001.70.5, 00000000002.70.5] 2020-11-19T03:22:10.818 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.106.5, , 00000000001.181.5, 00000000001.121.5, 00000000001.187.5, , 00000000001.76.5, 00000000001.88.5, 00000000001.70.5, 00000000002.70.5] 2020-11-19T03:22:10.819 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:10.811+0000 7fee871410c0 0 bucket checkpoint complete 2020-11-19T03:22:11.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:11.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:11.387 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:22:11.434 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:11.435 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:11.694 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:11.683+0000 7f61e5faa0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:22:11.694 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.101.5, 00000000002.106.5, , 00000000001.181.5, 00000000001.121.5, 00000000001.187.5, , 00000000001.76.5, 00000000001.88.5, 00000000001.70.5, 00000000002.70.5] 2020-11-19T03:22:11.694 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.101.5, 00000000003.108.5, , 00000000002.183.5, 00000000001.121.5, 00000000003.191.5, , 00000000003.80.5, 00000000002.90.5, 00000000002.72.5, 00000000003.72.5] 2020-11-19T03:22:16.698 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:16.691+0000 7f61e5faa0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:22:16.699 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.101.5, 00000000002.106.5, , 00000000001.181.5, 00000000001.121.5, 00000000001.187.5, , 00000000001.76.5, 00000000001.88.5, 00000000001.70.5, 00000000002.70.5] 2020-11-19T03:22:16.700 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.101.5, 00000000003.108.5, , 00000000002.183.5, 00000000001.121.5, 00000000003.191.5, , 00000000003.80.5, 00000000002.90.5, 00000000002.72.5, 00000000003.72.5] 2020-11-19T03:22:21.703 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:21.695+0000 7f61e5faa0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:22:21.703 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.101.5, 00000000002.106.5, , 00000000001.181.5, 00000000001.121.5, 00000000001.187.5, , 00000000001.76.5, 00000000001.88.5, 00000000001.70.5, 00000000002.70.5] 2020-11-19T03:22:21.704 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.101.5, 00000000003.108.5, , 00000000002.183.5, 00000000001.121.5, 00000000003.191.5, , 00000000003.80.5, 00000000002.90.5, 00000000002.72.5, 00000000003.72.5] 2020-11-19T03:22:26.707 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:26.699+0000 7f61e5faa0c0 1 bucket sync caught up with source: 2020-11-19T03:22:26.707 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.101.5, 00000000003.108.5, , 00000000002.183.5, 00000000001.121.5, 00000000003.191.5, , 00000000003.80.5, 00000000002.90.5, 00000000002.72.5, 00000000003.72.5] 2020-11-19T03:22:26.708 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.101.5, 00000000003.108.5, , 00000000002.183.5, 00000000001.121.5, 00000000003.191.5, , 00000000003.80.5, 00000000002.90.5, 00000000002.72.5, 00000000003.72.5] 2020-11-19T03:22:26.708 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:26.699+0000 7f61e5faa0c0 0 bucket checkpoint complete 2020-11-19T03:22:26.837 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2020-11-19T03:22:26.839 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2020-11-19T03:22:26.841 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2020-11-19T03:22:26.846 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2020-11-19T03:22:26.848 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2020-11-19T03:22:26.849 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2020-11-19T03:22:26.851 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2020-11-19T03:22:26.853 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2020-11-19T03:22:26.854 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2020-11-19T03:22:26.854 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2020-11-19T03:22:27.191 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... ok 2020-11-19T03:22:27.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:27.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:27.193 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:27.232 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:27.233 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:22:27.483 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:22:27.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:27.483 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756147.178243_1311.1", 2020-11-19T03:22:27.483 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:22:27.178243Z" 2020-11-19T03:22:27.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:27.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:27.484 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:27.484 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:27.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:27.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:27.485 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:27.485 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:27.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:27.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:27.486 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:27.486 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:27.486 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:27.486 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:22:27.487 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:22:27.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:27.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:27.489 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:27.532 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:27.532 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:27.800 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:22:27.801 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:22:27.801 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:27.801 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:22:27.801 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:22:27.802 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:22:27.802 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:27.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:27.802 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:22:27.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:27.803 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:22:27.803 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:27.803 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:27.803 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756147.178243_1311.1", 2020-11-19T03:22:27.803 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:27.804 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:22:27.804 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:27.804 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:22:27.178243Z", 2020-11-19T03:22:27.804 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:27.804 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:27.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:27.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:27.805 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:22:27.805 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:27.805 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:27.806 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:22:27.806 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:27.806 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:27.806 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:27.806 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:22:27.807 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:27.807 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:27.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:27.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:27.807 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:22:27.808 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:27.808 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:27.808 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:27.808 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:27.808 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:27.809 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:27.809 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:27.809 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:27.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:27.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:27.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:27.814 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:22:27.814 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:27.814 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:27.815 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:27.815 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:27.815 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:27.815 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:27.815 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:27.816 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:27.816 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:27.816 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:27.816 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:22:27.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:27.817 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:22:27.817 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:22:27.817 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:22:27.817 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:27.818 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:22:27.819 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:22:27.819 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:22:27.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:27.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:27.821 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:27.878 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:27.878 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:28.159 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:22:28.159 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:22:28.160 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:22:28.160 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:22:28.160 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:22:28.161 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:22:28.161 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:22:28.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:28.162 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:22:28.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:28.162 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:22:28.162 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:22:28.163 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:22:28.163 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756147.178243_1311.1", 2020-11-19T03:22:28.163 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:22:28.163 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:22:28.164 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:22:28.164 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:22:27.178243Z", 2020-11-19T03:22:28.164 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:22:28.164 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:28.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:28.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:28.165 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:22:28.166 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:22:28.166 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:22:28.166 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:22:28.166 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:22:28.167 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:22:28.167 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:22:28.167 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:22:28.168 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:22:28.168 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:28.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:28.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:28.169 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:22:28.169 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:22:28.169 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:22:28.169 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:28.170 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:22:28.170 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:22:28.170 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:22:28.171 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:22:28.171 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:22:28.171 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:28.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:28.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:28.172 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:22:28.173 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:22:28.173 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:22:28.173 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:28.173 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:22:28.174 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:22:28.174 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:22:28.174 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:22:28.175 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:22:28.175 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:28.175 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:28.175 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:22:28.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:28.176 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:22:28.176 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:22:28.176 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:22:28.177 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:28.177 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:22:28.178 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:22:28.178 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:22:28.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:28.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:28.181 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:28.223 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:28.224 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:28.593 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:22:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:22:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:22:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:22:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:22:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:22:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:22:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756147.178243_1311.1", 2020-11-19T03:22:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:22:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:22:27.178243Z", 2020-11-19T03:22:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:22:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:22:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:22:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:22:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:22:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:22:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:22:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:22:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:22:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:28.607 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:22:28.608 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:22:28.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:28.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:28.610 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:28.653 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:28.653 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:28.874 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:22:28.902 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:22:28.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:28.903 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756147.178243_1311.1", 2020-11-19T03:22:28.903 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:22:27.178243Z" 2020-11-19T03:22:28.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:28.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:28.904 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:28.904 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:28.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:28.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:28.904 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:28.905 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:28.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:28.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:28.905 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:28.906 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:28.906 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:28.906 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:22:28.907 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:22:28.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:28.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:28.909 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:28.950 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:28.950 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:29.322 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:22:29.323 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:22:29.323 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:29.324 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:22:29.324 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:22:29.324 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:22:29.325 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:29.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:29.326 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:22:29.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:29.326 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:22:29.327 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:29.327 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:29.327 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756147.178243_1311.1", 2020-11-19T03:22:29.328 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:29.328 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:22:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:22:27.178243Z", 2020-11-19T03:22:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:22:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:22:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:29.331 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:22:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:29.332 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:22:29.333 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:29.333 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:29.333 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:29.333 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:29.333 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:29.334 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:29.334 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:29.334 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:29.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:29.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:29.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:29.335 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:22:29.335 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:29.335 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:29.335 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:29.336 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:29.337 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:29.337 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:29.337 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:22:29.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:29.337 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:22:29.338 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:22:29.338 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:22:29.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:29.338 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:22:29.340 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:22:29.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:29.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:29.438 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:29.482 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:29.482 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:29.738 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:22:29.773 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:22:29.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:29.773 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756149.431015_1329.1", 2020-11-19T03:22:29.774 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:22:29.431015Z" 2020-11-19T03:22:29.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:29.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:29.774 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:29.775 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:29.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:29.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:29.775 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:29.775 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:29.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:22:29.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:22:29.776 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:22:29.776 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:22:29.776 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:22:29.777 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:22:29.777 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:22:29.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:29.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:22:29.779 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:22:29.820 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:29.821 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:30.186 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:22:30.187 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:22:30.187 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:22:30.187 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:22:30.188 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:22:30.188 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:22:30.188 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:30.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:30.188 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:22:30.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:30.189 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:22:30.189 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:30.189 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:30.189 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756149.431015_1329.1", 2020-11-19T03:22:30.190 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:30.190 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:22:30.190 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:30.190 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:22:29.431015Z", 2020-11-19T03:22:30.191 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:22:30.191 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:30.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:30.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:30.191 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:22:30.192 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:30.192 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:30.192 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:22:30.192 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:30.192 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:30.193 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:30.193 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:22:30.193 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:30.193 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:30.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:30.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:30.194 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:22:30.194 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:30.194 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:30.195 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:30.195 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:30.195 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:30.195 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:30.195 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:30.196 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:30.196 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:30.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:30.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:22:30.197 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:22:30.197 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:22:30.197 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:22:30.197 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:22:30.197 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:22:30.198 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:22:30.198 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:22:30.198 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:22:30.199 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:22:30.199 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:30.199 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:30.199 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:22:30.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:22:30.200 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:22:30.200 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:22:30.200 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:22:30.200 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:22:30.200 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:22:30.202 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:22:30.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:30.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:30.203 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:22:30.293 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:30.293 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:30.547 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:30.539+0000 7fb285c870c0 1 bucket sync caught up with empty source 2020-11-19T03:22:30.548 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:30.539+0000 7fb285c870c0 0 bucket checkpoint complete 2020-11-19T03:22:31.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:31.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:31.164 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:22:31.219 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:31.219 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:31.509 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:31.499+0000 7f680871c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:22:31.510 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.54.5, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000001.146.5] 2020-11-19T03:22:31.510 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.57.3, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000002.148.5] 2020-11-19T03:22:36.514 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:36.503+0000 7f680871c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:22:36.514 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.54.5, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000001.146.5] 2020-11-19T03:22:36.514 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.57.3, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000002.148.5] 2020-11-19T03:22:41.518 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:41.511+0000 7f680871c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:22:41.518 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.54.5, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000001.146.5] 2020-11-19T03:22:41.518 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.57.3, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000002.148.5] 2020-11-19T03:22:46.523 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:46.515+0000 7f680871c0c0 1 bucket sync caught up with source: 2020-11-19T03:22:46.524 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.57.3, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000002.148.5] 2020-11-19T03:22:46.524 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.57.3, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000002.148.5] 2020-11-19T03:22:46.524 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:46.515+0000 7f680871c0c0 0 bucket checkpoint complete 2020-11-19T03:22:46.704 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2020-11-19T03:22:46.705 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2020-11-19T03:22:46.705 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2020-11-19T03:22:46.706 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2020-11-19T03:22:46.706 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2020-11-19T03:22:46.706 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2020-11-19T03:22:46.707 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2020-11-19T03:22:46.707 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2020-11-19T03:22:46.708 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:22:46.708 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:22:46.708 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2020-11-19T03:22:46.718 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-19T03:22:46.734 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2020-11-19T03:22:46.735 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2020-11-19T03:22:46.735 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2020-11-19T03:22:46.736 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2020-11-19T03:22:46.736 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2020-11-19T03:22:46.736 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2020-11-19T03:22:46.737 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2020-11-19T03:22:46.737 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2020-11-19T03:22:46.737 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:22:46.738 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:22:46.738 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2020-11-19T03:22:46.739 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-19T03:22:46.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:46.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:22:46.861 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:22:46.910 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:22:46.910 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:22:47.149 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:47.135+0000 7f705bd6d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:22:47.149 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.57.3, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000002.148.5] 2020-11-19T03:22:47.149 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.63.5, , 00000000002.59.5, 00000000002.192.5, 00000000002.106.5, , 00000000002.160.5, 00000000002.119.5, 00000000002.158.5, 00000000004.152.5] 2020-11-19T03:22:52.150 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:52.139+0000 7f705bd6d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:22:52.150 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.57.3, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000002.148.5] 2020-11-19T03:22:52.150 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.63.5, , 00000000002.59.5, 00000000002.192.5, 00000000002.106.5, , 00000000002.160.5, 00000000002.119.5, 00000000002.158.5, 00000000004.152.5] 2020-11-19T03:22:57.154 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:22:57.143+0000 7f705bd6d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:22:57.154 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.57.3, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000002.148.5] 2020-11-19T03:22:57.154 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.63.5, , 00000000002.59.5, 00000000002.192.5, 00000000002.106.5, , 00000000002.160.5, 00000000002.119.5, 00000000002.158.5, 00000000004.152.5] 2020-11-19T03:23:02.158 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:02.151+0000 7f705bd6d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:23:02.159 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.57.3, , 00000000001.52.5, 00000000001.190.5, 00000000001.104.5, , 00000000001.156.5, 00000000001.117.5, 00000000001.154.5, 00000000002.148.5] 2020-11-19T03:23:02.159 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.63.5, , 00000000002.59.5, 00000000002.192.5, 00000000002.106.5, , 00000000002.160.5, 00000000002.119.5, 00000000002.158.5, 00000000004.152.5] 2020-11-19T03:23:07.164 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:07.155+0000 7f705bd6d0c0 1 bucket sync caught up with source: 2020-11-19T03:23:07.165 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000005.63.5, , 00000000002.59.5, 00000000002.192.5, 00000000002.106.5, , 00000000002.160.5, 00000000002.119.5, 00000000002.158.5, 00000000004.152.5] 2020-11-19T03:23:07.165 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000005.63.5, , 00000000002.59.5, 00000000002.192.5, 00000000002.106.5, , 00000000002.160.5, 00000000002.119.5, 00000000002.158.5, 00000000004.152.5] 2020-11-19T03:23:07.166 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:07.155+0000 7f705bd6d0c0 0 bucket checkpoint complete 2020-11-19T03:23:07.295 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE) synced 2020-11-19T03:23:07.308 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2020-11-19T03:23:07.309 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2020-11-19T03:23:07.309 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2020-11-19T03:23:07.310 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2020-11-19T03:23:07.311 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2020-11-19T03:23:07.311 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2020-11-19T03:23:07.312 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2020-11-19T03:23:07.313 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2020-11-19T03:23:07.314 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:23:07.314 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:23:07.315 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2020-11-19T03:23:07.316 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-19T03:23:07.328 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "4" type: "OBJECT_DELETE" 2020-11-19T03:23:07.329 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "8" type: "OBJECT_DELETE" 2020-11-19T03:23:07.329 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "6" type: "OBJECT_DELETE" 2020-11-19T03:23:07.330 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "7" type: "OBJECT_DELETE" 2020-11-19T03:23:07.330 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "1" type: "OBJECT_DELETE" 2020-11-19T03:23:07.330 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "2" type: "OBJECT_DELETE" 2020-11-19T03:23:07.331 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "0" type: "OBJECT_DELETE" 2020-11-19T03:23:07.331 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "3" type: "OBJECT_DELETE" 2020-11-19T03:23:07.331 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "5" type: "OBJECT_DELETE" 2020-11-19T03:23:07.332 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "9" type: "OBJECT_DELETE" 2020-11-19T03:23:07.332 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-19T03:23:07.347 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2020-11-19T03:23:07.348 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2020-11-19T03:23:07.348 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2020-11-19T03:23:07.349 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2020-11-19T03:23:07.350 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2020-11-19T03:23:07.351 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2020-11-19T03:23:07.352 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2020-11-19T03:23:07.353 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2020-11-19T03:23:07.354 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:23:07.354 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:23:07.354 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2020-11-19T03:23:07.355 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-19T03:23:07.517 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... ok 2020-11-19T03:23:07.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:07.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:07.519 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:07.558 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:07.786 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:23:07.819 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:23:07.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:07.820 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756187.508254_1342.1", 2020-11-19T03:23:07.820 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:23:07.508254Z" 2020-11-19T03:23:07.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:07.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:07.821 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:07.821 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:07.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:07.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:07.822 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:07.822 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:07.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:07.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:07.823 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:07.823 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:07.823 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:07.823 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:23:07.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:23:07.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:07.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:07.825 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:07.869 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:07.870 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:08.146 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:23:08.146 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:23:08.147 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:23:08.147 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:23:08.147 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:23:08.147 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:08.148 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:08.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.148 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:23:08.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:08.149 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:23:08.149 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:08.149 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:08.149 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756187.508254_1342.1", 2020-11-19T03:23:08.149 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:08.150 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:23:08.150 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:08.150 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:23:07.508254Z", 2020-11-19T03:23:08.150 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:08.150 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:08.151 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:23:08.151 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:08.152 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:08.152 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:08.152 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:08.152 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:08.152 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:08.153 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:08.153 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:08.153 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:08.154 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:23:08.154 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:08.154 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:08.154 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:08.154 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:08.155 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:08.155 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:08.155 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:08.156 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:08.156 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:08.156 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:23:08.157 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:08.157 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:08.157 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:08.157 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:08.158 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:08.158 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:08.158 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:08.158 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:08.159 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.159 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.159 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:23:08.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.159 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:23:08.160 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:23:08.160 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:23:08.160 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.160 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:23:08.161 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:23:08.162 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:23:08.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:08.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:08.163 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:08.196 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:08.196 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:08.465 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:23:08.465 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:23:08.466 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:23:08.466 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:23:08.466 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:23:08.466 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:08.467 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:23:08.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:08.467 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:23:08.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:08.468 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:23:08.468 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:08.468 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:08.468 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756187.508254_1342.1", 2020-11-19T03:23:08.468 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:08.469 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:23:08.469 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:08.469 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:23:07.508254Z", 2020-11-19T03:23:08.469 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:23:08.469 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:08.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:08.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:08.471 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:23:08.472 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:08.472 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:08.472 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:08.472 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:08.473 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:23:08.473 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:08.473 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:08.473 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:23:08.474 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:08.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:08.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:08.474 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:23:08.475 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:08.475 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:08.475 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:08.475 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:08.475 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:23:08.475 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:08.476 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:23:08.476 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:23:08.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:08.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:08.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:08.477 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:23:08.477 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:08.477 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:08.477 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:08.477 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:08.478 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:23:08.478 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:08.478 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:23:08.478 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:23:08.478 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:08.479 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:08.479 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:23:08.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:08.479 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:23:08.480 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:23:08.480 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:23:08.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:08.481 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:23:08.482 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:23:08.482 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:23:08.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:08.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:08.490 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:08.543 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:08.543 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:08.915 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:23:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:23:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:23:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:23:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:23:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:23:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:23:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756187.508254_1342.1", 2020-11-19T03:23:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:23:08.929 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:23:07.508254Z", 2020-11-19T03:23:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:23:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:23:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:23:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:08.944 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:08.944 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:23:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:23:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:23:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:23:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:08.947 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:23:08.948 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:23:08.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:08.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:08.950 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:08.995 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:08.995 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:09.223 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:23:09.259 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:23:09.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:09.260 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756187.508254_1342.1", 2020-11-19T03:23:09.261 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:23:07.508254Z" 2020-11-19T03:23:09.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:09.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:09.262 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:09.262 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:09.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:09.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:09.264 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:09.264 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:09.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:09.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:09.265 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:09.266 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:09.266 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:09.266 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:23:09.267 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:23:09.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:09.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:09.269 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:09.315 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:09.315 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:09.571 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:23:09.571 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:23:09.571 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:23:09.571 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:23:09.572 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:23:09.572 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:09.572 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:09.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:09.572 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:23:09.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:09.573 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:23:09.573 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:09.574 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:09.575 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756187.508254_1342.1", 2020-11-19T03:23:09.576 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:09.576 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:23:09.577 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:09.577 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:23:07.508254Z", 2020-11-19T03:23:09.580 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:09.581 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:09.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:09.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:09.583 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:23:09.584 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:09.584 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:09.585 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:09.586 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:09.587 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:09.588 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:09.588 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:09.589 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:09.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:09.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:09.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:09.592 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:23:09.595 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:09.595 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:09.597 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:09.597 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:09.598 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:09.598 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:09.599 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:09.599 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:09.600 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:09.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:09.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:09.601 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:23:09.601 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:09.602 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:09.603 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:09.603 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:09.604 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:09.605 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:09.607 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:09.608 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:09.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:09.609 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:09.609 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:23:09.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:09.610 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:23:09.611 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:23:09.612 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:23:09.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:09.613 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:23:09.616 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:23:09.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:09.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:09.681 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:09.730 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:09.730 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:23:09.977 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:23:09.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:09.979 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756189.673224_1352.1", 2020-11-19T03:23:09.979 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:23:09.673224Z" 2020-11-19T03:23:09.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:09.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:09.980 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:09.980 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:09.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:09.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:09.982 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:09.982 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:09.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:09.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:09.983 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:09.984 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:09.984 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:09.984 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:23:09.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:23:09.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:09.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:09.989 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:10.032 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:10.032 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:10.298 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:23:10.299 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:23:10.299 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:23:10.300 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:23:10.300 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:23:10.300 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:10.301 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:10.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:10.302 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:23:10.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:10.302 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:23:10.303 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:10.303 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:10.304 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756189.673224_1352.1", 2020-11-19T03:23:10.304 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:10.304 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:23:10.305 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:10.305 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:23:09.673224Z", 2020-11-19T03:23:10.306 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:10.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:10.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:10.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:10.308 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:23:10.308 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:10.309 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:10.309 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:10.309 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:10.310 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:10.310 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:10.311 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:10.311 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:10.311 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:10.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:10.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:10.313 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:23:10.313 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:10.313 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:10.314 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:10.315 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:10.315 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:10.316 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:10.316 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:10.316 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:10.317 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:10.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:10.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:10.318 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:23:10.318 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:10.319 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:10.319 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:10.320 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:10.320 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:10.320 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:10.321 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:10.321 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:10.321 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:10.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:10.322 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:23:10.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:10.323 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:23:10.323 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:23:10.324 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:23:10.324 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:10.325 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:23:10.326 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:23:15.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-85', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:23:15.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-85', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:23:15.242 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-85 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:23:15.291 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:15.291 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:15.542 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:15.531+0000 7f66c4def0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:23:15.543 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.200.5, 00000000002.132.5, 00000000001.173.5, 00000000001.165.5, 00000000001.167.5, 00000000001.81.5, 00000000001.97.5, 00000000001.109.5, 00000000001.106.5, 00000000001.122.5, 00000000001.88.5] 2020-11-19T03:23:15.543 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000010.218.5, 00000000010.148.5, 00000000009.215.5, 00000000010.211.5, 00000000007.203.5, 00000000007.93.5, 00000000008.111.5, 00000000010.127.5, 00000000010.124.5, 00000000010.140.5, 00000000009.104.5] 2020-11-19T03:23:20.544 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:20.539+0000 7f66c4def0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:23:20.545 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.200.5, 00000000002.132.5, 00000000001.173.5, 00000000001.165.5, 00000000001.167.5, 00000000001.81.5, 00000000001.97.5, 00000000001.109.5, 00000000001.106.5, 00000000001.122.5, 00000000001.88.5] 2020-11-19T03:23:20.545 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000010.218.5, 00000000010.148.5, 00000000009.215.5, 00000000010.211.5, 00000000007.203.5, 00000000007.93.5, 00000000008.111.5, 00000000010.127.5, 00000000010.124.5, 00000000010.140.5, 00000000009.104.5] 2020-11-19T03:23:25.549 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:25.543+0000 7f66c4def0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:23:25.549 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.200.5, 00000000002.132.5, 00000000001.173.5, 00000000001.165.5, 00000000001.167.5, 00000000001.81.5, 00000000001.97.5, 00000000001.109.5, 00000000001.106.5, 00000000001.122.5, 00000000001.88.5] 2020-11-19T03:23:25.550 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000010.218.5, 00000000010.148.5, 00000000009.215.5, 00000000010.211.5, 00000000007.203.5, 00000000007.93.5, 00000000008.111.5, 00000000010.127.5, 00000000010.124.5, 00000000010.140.5, 00000000009.104.5] 2020-11-19T03:23:30.553 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:30.547+0000 7f66c4def0c0 1 bucket sync caught up with source: 2020-11-19T03:23:30.554 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000010.218.5, 00000000010.148.5, 00000000009.215.5, 00000000010.211.5, 00000000007.203.5, 00000000007.93.5, 00000000008.111.5, 00000000010.127.5, 00000000010.124.5, 00000000010.140.5, 00000000009.104.5] 2020-11-19T03:23:30.554 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000010.218.5, 00000000010.148.5, 00000000009.215.5, 00000000010.211.5, 00000000007.203.5, 00000000007.93.5, 00000000008.111.5, 00000000010.127.5, 00000000010.124.5, 00000000010.140.5, 00000000009.104.5] 2020-11-19T03:23:30.554 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:30.547+0000 7f66c4def0c0 0 bucket checkpoint complete 2020-11-19T03:23:30.690 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2020-11-19T03:23:30.691 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2020-11-19T03:23:30.691 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2020-11-19T03:23:30.692 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2020-11-19T03:23:30.692 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2020-11-19T03:23:30.693 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2020-11-19T03:23:30.693 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-11-19T03:23:30.693 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2020-11-19T03:23:30.694 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2020-11-19T03:23:30.694 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2020-11-19T03:23:30.695 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "24" type: "OBJECT_CREATE" 2020-11-19T03:23:30.695 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "40" type: "OBJECT_CREATE" 2020-11-19T03:23:30.695 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2020-11-19T03:23:30.696 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-11-19T03:23:30.696 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "50" type: "OBJECT_CREATE" 2020-11-19T03:23:30.707 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "54" type: "OBJECT_CREATE" 2020-11-19T03:23:30.708 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "21" type: "OBJECT_CREATE" 2020-11-19T03:23:30.708 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "36" type: "OBJECT_CREATE" 2020-11-19T03:23:30.709 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "69" type: "OBJECT_CREATE" 2020-11-19T03:23:30.709 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "32" type: "OBJECT_CREATE" 2020-11-19T03:23:30.709 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "77" type: "OBJECT_CREATE" 2020-11-19T03:23:30.710 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "55" type: "OBJECT_CREATE" 2020-11-19T03:23:30.711 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "95" type: "OBJECT_CREATE" 2020-11-19T03:23:30.711 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "59" type: "OBJECT_CREATE" 2020-11-19T03:23:30.711 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "96" type: "OBJECT_CREATE" 2020-11-19T03:23:30.712 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "73" type: "OBJECT_CREATE" 2020-11-19T03:23:30.712 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "91" type: "OBJECT_CREATE" 2020-11-19T03:23:30.713 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "57" type: "OBJECT_CREATE" 2020-11-19T03:23:30.713 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "63" type: "OBJECT_CREATE" 2020-11-19T03:23:30.714 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "28" type: "OBJECT_CREATE" 2020-11-19T03:23:30.722 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "42" type: "OBJECT_CREATE" 2020-11-19T03:23:30.723 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2020-11-19T03:23:30.723 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "75" type: "OBJECT_CREATE" 2020-11-19T03:23:30.724 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2020-11-19T03:23:30.724 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "47" type: "OBJECT_CREATE" 2020-11-19T03:23:30.725 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "83" type: "OBJECT_CREATE" 2020-11-19T03:23:30.725 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "39" type: "OBJECT_CREATE" 2020-11-19T03:23:30.725 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "92" type: "OBJECT_CREATE" 2020-11-19T03:23:30.726 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "78" type: "OBJECT_CREATE" 2020-11-19T03:23:30.726 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "93" type: "OBJECT_CREATE" 2020-11-19T03:23:30.727 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "62" type: "OBJECT_CREATE" 2020-11-19T03:23:30.727 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "64" type: "OBJECT_CREATE" 2020-11-19T03:23:30.728 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "25" type: "OBJECT_CREATE" 2020-11-19T03:23:30.728 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "46" type: "OBJECT_CREATE" 2020-11-19T03:23:30.729 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "60" type: "OBJECT_CREATE" 2020-11-19T03:23:30.741 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "82" type: "OBJECT_CREATE" 2020-11-19T03:23:30.742 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "44" type: "OBJECT_CREATE" 2020-11-19T03:23:30.742 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "66" type: "OBJECT_CREATE" 2020-11-19T03:23:30.743 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "79" type: "OBJECT_CREATE" 2020-11-19T03:23:30.744 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "48" type: "OBJECT_CREATE" 2020-11-19T03:23:30.744 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "26" type: "OBJECT_CREATE" 2020-11-19T03:23:30.745 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "80" type: "OBJECT_CREATE" 2020-11-19T03:23:30.745 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "98" type: "OBJECT_CREATE" 2020-11-19T03:23:30.745 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "33" type: "OBJECT_CREATE" 2020-11-19T03:23:30.746 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "97" type: "OBJECT_CREATE" 2020-11-19T03:23:30.746 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "65" type: "OBJECT_CREATE" 2020-11-19T03:23:30.747 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "88" type: "OBJECT_CREATE" 2020-11-19T03:23:30.747 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "61" type: "OBJECT_CREATE" 2020-11-19T03:23:30.747 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "29" type: "OBJECT_CREATE" 2020-11-19T03:23:30.748 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "43" type: "OBJECT_CREATE" 2020-11-19T03:23:30.757 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "72" type: "OBJECT_CREATE" 2020-11-19T03:23:30.758 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "37" type: "OBJECT_CREATE" 2020-11-19T03:23:30.758 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "22" type: "OBJECT_CREATE" 2020-11-19T03:23:30.758 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "67" type: "OBJECT_CREATE" 2020-11-19T03:23:30.759 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2020-11-19T03:23:30.759 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "94" type: "OBJECT_CREATE" 2020-11-19T03:23:30.759 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "87" type: "OBJECT_CREATE" 2020-11-19T03:23:30.760 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2020-11-19T03:23:30.761 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "49" type: "OBJECT_CREATE" 2020-11-19T03:23:30.761 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "76" type: "OBJECT_CREATE" 2020-11-19T03:23:30.761 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "51" type: "OBJECT_CREATE" 2020-11-19T03:23:30.762 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "90" type: "OBJECT_CREATE" 2020-11-19T03:23:30.762 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "34" type: "OBJECT_CREATE" 2020-11-19T03:23:30.762 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2020-11-19T03:23:30.763 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "52" type: "OBJECT_CREATE" 2020-11-19T03:23:30.771 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "70" type: "OBJECT_CREATE" 2020-11-19T03:23:30.772 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "45" type: "OBJECT_CREATE" 2020-11-19T03:23:30.772 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "89" type: "OBJECT_CREATE" 2020-11-19T03:23:30.773 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2020-11-19T03:23:30.773 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "81" type: "OBJECT_CREATE" 2020-11-19T03:23:30.773 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "30" type: "OBJECT_CREATE" 2020-11-19T03:23:30.774 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "84" type: "OBJECT_CREATE" 2020-11-19T03:23:30.774 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2020-11-19T03:23:30.775 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "99" type: "OBJECT_CREATE" 2020-11-19T03:23:30.775 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "31" type: "OBJECT_CREATE" 2020-11-19T03:23:30.775 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "53" type: "OBJECT_CREATE" 2020-11-19T03:23:30.776 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "35" type: "OBJECT_CREATE" 2020-11-19T03:23:30.776 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "20" type: "OBJECT_CREATE" 2020-11-19T03:23:30.776 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "68" type: "OBJECT_CREATE" 2020-11-19T03:23:30.777 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "85" type: "OBJECT_CREATE" 2020-11-19T03:23:30.793 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2020-11-19T03:23:30.794 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "86" type: "OBJECT_CREATE" 2020-11-19T03:23:30.794 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "71" type: "OBJECT_CREATE" 2020-11-19T03:23:30.795 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "27" type: "OBJECT_CREATE" 2020-11-19T03:23:30.795 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "38" type: "OBJECT_CREATE" 2020-11-19T03:23:30.795 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "41" type: "OBJECT_CREATE" 2020-11-19T03:23:30.796 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "23" type: "OBJECT_CREATE" 2020-11-19T03:23:30.796 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "74" type: "OBJECT_CREATE" 2020-11-19T03:23:30.796 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "56" type: "OBJECT_CREATE" 2020-11-19T03:23:30.797 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "58" type: "OBJECT_CREATE" 2020-11-19T03:23:30.813 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-19T03:23:32.140 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... ok 2020-11-19T03:23:32.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:32.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:32.142 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:32.182 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:32.183 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:32.390 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:23:32.422 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:23:32.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:32.422 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756212.130289_1357.1", 2020-11-19T03:23:32.423 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:23:32.130289Z" 2020-11-19T03:23:32.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:32.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:32.424 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:32.424 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:32.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:32.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:32.425 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:32.425 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:32.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:32.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:32.426 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:32.426 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:32.427 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:32.427 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:23:32.429 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:23:32.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:32.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:32.431 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:32.478 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:32.479 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:32.727 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:23:32.727 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:23:32.728 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:23:32.728 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:23:32.728 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:23:32.729 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:32.729 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:32.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:32.731 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:23:32.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:32.731 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:23:32.731 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:32.732 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:32.732 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756212.130289_1357.1", 2020-11-19T03:23:32.732 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:32.732 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:23:32.732 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:32.733 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:23:32.130289Z", 2020-11-19T03:23:32.733 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:32.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:32.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:32.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:32.734 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:23:32.734 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:32.734 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:32.734 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:32.734 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:32.735 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:32.735 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:32.735 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:32.735 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:32.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:32.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:32.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:32.736 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:23:32.736 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:32.737 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:32.737 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:32.737 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:32.737 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:32.738 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:32.738 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:32.738 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:32.738 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:32.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:32.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:32.739 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:23:32.739 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:32.739 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:32.739 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:32.740 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:32.740 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:32.740 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:32.740 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:32.741 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:32.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:32.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:32.741 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:23:32.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:32.742 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:23:32.742 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:23:32.742 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:23:32.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:32.742 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:23:32.744 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:23:32.744 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:23:32.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:32.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:32.746 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:32.795 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:32.795 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:33.052 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:23:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:23:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:23:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:23:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:23:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:23:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:23:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:23:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756212.130289_1357.1", 2020-11-19T03:23:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:23:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:23:32.130289Z", 2020-11-19T03:23:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:23:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:23:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:23:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:23:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:23:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:23:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:23:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:23:33.062 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:33.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:33.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:33.062 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:23:33.062 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:33.063 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:33.063 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:33.063 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:33.063 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:23:33.063 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:33.064 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:23:33.064 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:23:33.064 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:33.064 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:33.064 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:23:33.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:33.065 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:23:33.065 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:23:33.065 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:23:33.066 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:33.066 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:23:33.067 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:23:33.067 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:23:33.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:33.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:33.069 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:33.111 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:33.112 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:33.480 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:23:33.481 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:23:33.481 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:23:33.481 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:23:33.481 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:23:33.482 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:33.482 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:33.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:33.482 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:23:33.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:33.483 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:23:33.483 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:33.483 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:33.483 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756212.130289_1357.1", 2020-11-19T03:23:33.484 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:33.484 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:23:33.484 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:33.484 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:23:32.130289Z", 2020-11-19T03:23:33.484 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:33.485 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:33.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:33.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:33.485 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:23:33.486 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:33.486 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:33.487 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:33.487 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:33.487 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:33.487 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:33.489 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:33.489 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:33.490 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:33.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:33.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:33.491 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:23:33.491 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:33.491 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:33.491 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:33.491 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:33.492 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:33.492 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:33.492 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:33.492 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:33.492 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:33.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:33.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:33.493 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:23:33.493 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:33.493 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:33.494 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:33.494 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:33.494 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:33.494 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:33.495 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:33.495 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:33.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:33.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:33.495 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:23:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:23:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:23:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:23:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:33.497 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:23:33.498 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:23:33.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:33.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:33.500 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:33.541 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:33.542 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:33.796 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:23:33.827 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:23:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756212.130289_1357.1", 2020-11-19T03:23:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:23:32.130289Z" 2020-11-19T03:23:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:33.831 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:23:33.832 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:23:33.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:33.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:33.833 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:33.873 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:33.874 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:34.249 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:23:34.250 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:23:34.250 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:23:34.250 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:23:34.250 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:23:34.251 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:34.251 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:34.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:34.251 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:23:34.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:34.252 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:23:34.252 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:34.252 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:34.252 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756212.130289_1357.1", 2020-11-19T03:23:34.252 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:34.253 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:23:34.253 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:34.253 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:23:32.130289Z", 2020-11-19T03:23:34.253 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:34.254 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:34.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:34.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:34.254 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:23:34.254 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:34.255 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:34.255 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:34.255 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:34.255 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:34.255 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:34.257 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:23:34.257 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:34.257 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:34.257 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:34.257 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:34.258 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:34.258 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:34.258 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:34.258 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:34.258 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:34.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:34.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:34.259 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:23:34.259 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:34.259 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:34.260 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:34.260 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:34.260 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:34.260 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:34.260 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:34.261 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:34.261 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:34.261 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:34.261 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:23:34.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:34.262 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:23:34.262 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:23:34.262 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:23:34.262 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:34.262 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:23:34.264 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:23:34.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:34.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:34.339 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:34.604 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:23:34.639 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:23:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756214.331558_1367.1", 2020-11-19T03:23:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:23:34.331558Z" 2020-11-19T03:23:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:34.643 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:23:34.644 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:23:34.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:34.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:34.646 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:34.686 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:34.686 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:35.055 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:23:35.055 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:23:35.056 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:23:35.056 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:23:35.056 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:23:35.057 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:35.057 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:35.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:35.058 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:23:35.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:35.058 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:23:35.058 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:35.059 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:35.059 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756214.331558_1367.1", 2020-11-19T03:23:35.059 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:35.060 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:23:35.060 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:35.060 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:23:34.331558Z", 2020-11-19T03:23:35.061 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:35.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:35.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:35.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:35.062 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:23:35.062 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:35.063 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:35.063 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:35.063 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:35.064 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:35.064 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:35.064 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:35.064 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:35.065 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:35.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:35.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:35.066 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:23:35.066 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:35.066 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:35.067 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:35.067 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:35.067 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:35.067 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:35.068 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:35.068 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:35.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:35.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:35.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:35.069 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:23:35.070 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:35.070 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:35.070 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:35.071 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:35.071 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:35.071 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:35.071 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:35.072 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:35.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:35.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:35.073 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:23:35.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:35.073 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:23:35.074 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:23:35.074 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:23:35.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:35.074 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:23:35.076 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:23:35.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:23:35.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'atsuux-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:23:35.569 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket atsuux-86 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:23:35.617 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:35.618 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:35.843 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:35.838+0000 7fadfb0260c0 1 waiting for incremental sync to catch up: 2020-11-19T03:23:35.843 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.110.5, , 00000000001.88.5, 00000000001.56.5, 00000000001.109.5, , 00000000001.67.5, 00000000001.147.5, 00000000001.141.5, 00000000001.143.5] 2020-11-19T03:23:35.843 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.112.5, , 00000000001.88.5, 00000000001.56.5, 00000000001.109.5, , 00000000001.67.5, 00000000001.147.5, 00000000001.141.5, 00000000002.145.5] 2020-11-19T03:23:40.848 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:40.842+0000 7fadfb0260c0 1 waiting for incremental sync to catch up: 2020-11-19T03:23:40.848 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.110.5, , 00000000001.88.5, 00000000001.56.5, 00000000001.109.5, , 00000000001.67.5, 00000000001.147.5, 00000000001.141.5, 00000000001.143.5] 2020-11-19T03:23:40.849 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.112.5, , 00000000001.88.5, 00000000001.56.5, 00000000001.109.5, , 00000000001.67.5, 00000000001.147.5, 00000000001.141.5, 00000000002.145.5] 2020-11-19T03:23:45.852 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:45.846+0000 7fadfb0260c0 1 waiting for incremental sync to catch up: 2020-11-19T03:23:45.852 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.110.5, , 00000000001.88.5, 00000000001.56.5, 00000000001.109.5, , 00000000001.67.5, 00000000001.147.5, 00000000001.141.5, 00000000001.143.5] 2020-11-19T03:23:45.852 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.112.5, , 00000000001.88.5, 00000000001.56.5, 00000000001.109.5, , 00000000001.67.5, 00000000001.147.5, 00000000001.141.5, 00000000002.145.5] 2020-11-19T03:23:50.855 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:50.850+0000 7fadfb0260c0 1 bucket sync caught up with source: 2020-11-19T03:23:50.856 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.112.5, , 00000000001.88.5, 00000000001.56.5, 00000000001.109.5, , 00000000001.67.5, 00000000001.147.5, 00000000001.141.5, 00000000002.145.5] 2020-11-19T03:23:50.856 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.112.5, , 00000000001.88.5, 00000000001.56.5, 00000000001.109.5, , 00000000001.67.5, 00000000001.147.5, 00000000001.141.5, 00000000002.145.5] 2020-11-19T03:23:50.856 INFO:teuthology.orchestra.run.smithi160.stderr:2020-11-19T03:23:50.850+0000 7fadfb0260c0 0 bucket checkpoint complete 2020-11-19T03:23:51.004 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756215.249683.5c0445fd" 2020-11-19T03:23:51.008 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756215.291732.af469c86" 2020-11-19T03:23:51.009 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756215.439137.3fe330ca" 2020-11-19T03:23:51.009 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756215.484128.8473a943" 2020-11-19T03:23:51.010 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756215.496349.30209157" 2020-11-19T03:23:51.010 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756215.535720.acc2415e" 2020-11-19T03:23:51.010 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756215.695786.9bca0bf3" 2020-11-19T03:23:51.011 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756215.743705.789906ae" 2020-11-19T03:23:51.011 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756227.990104.d76985f8" 2020-11-19T03:23:51.011 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1605756228.055768.5ba8bf21" 2020-11-19T03:23:51.022 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-19T03:23:51.107 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1605756215.535720.acc2415e" 2020-11-19T03:23:51.107 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1605756215.695786.9bca0bf3" 2020-11-19T03:23:51.108 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1605756215.743705.789906ae" 2020-11-19T03:23:51.108 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1605756227.990104.d76985f8" 2020-11-19T03:23:51.108 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1605756228.055768.5ba8bf21" 2020-11-19T03:23:51.302 INFO:tasks.rgw_multisite_tests:test acking of some events in a subscription ... ok 2020-11-19T03:23:51.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:51.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:51.306 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:51.349 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:51.350 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:51.593 INFO:teuthology.orchestra.run.smithi086.stderr:No --period given, using current period=43cc2be5-65c1-4813-aee6-d404c9e85e90 2020-11-19T03:23:51.624 INFO:teuthology.orchestra.run.smithi086.stdout:[ 2020-11-19T03:23:51.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:51.625 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756231.294848_1375.1", 2020-11-19T03:23:51.626 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "2020-11-19T03:23:51.294848Z" 2020-11-19T03:23:51.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:51.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:51.628 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:51.629 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:51.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:51.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:51.631 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:51.631 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:51.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:51.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:51.632 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:51.633 INFO:teuthology.orchestra.run.smithi086.stdout: "last_update": "0.000000" 2020-11-19T03:23:51.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:51.633 INFO:teuthology.orchestra.run.smithi086.stdout:] 2020-11-19T03:23:51.635 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:23:51.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:51.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:51.637 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:51.680 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:51.681 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:51.962 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2020-11-19T03:23:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2020-11-19T03:23:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2020-11-19T03:23:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2020-11-19T03:23:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2020-11-19T03:23:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2020-11-19T03:23:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2020-11-19T03:23:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605756231.294848_1375.1", 2020-11-19T03:23:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2020-11-19T03:23:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T03:23:51.294848Z", 2020-11-19T03:23:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2020-11-19T03:23:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2020-11-19T03:23:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2020-11-19T03:23:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2020-11-19T03:23:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2020-11-19T03:23:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2020-11-19T03:23:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2020-11-19T03:23:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2020-11-19T03:23:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2020-11-19T03:23:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2020-11-19T03:23:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2020-11-19T03:23:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2020-11-19T03:23:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2020-11-19T03:23:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2020-11-19T03:23:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2020-11-19T03:23:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2020-11-19T03:23:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2020-11-19T03:23:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2020-11-19T03:23:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2020-11-19T03:23:51.976 INFO:teuthology.orchestra.run.smithi160.stdout:} 2020-11-19T03:23:51.977 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:23:51.978 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:23:51.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:51.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:51.980 INFO:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:52.021 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:52.021 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:23:52.294 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2020-11-19T03:23:52.295 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_status": { 2020-11-19T03:23:52.295 INFO:teuthology.orchestra.run.smithi086.stdout: "info": { 2020-11-19T03:23:52.295 INFO:teuthology.orchestra.run.smithi086.stdout: "status": "sync", 2020-11-19T03:23:52.296 INFO:teuthology.orchestra.run.smithi086.stdout: "num_shards": 4, 2020-11-19T03:23:52.296 INFO:teuthology.orchestra.run.smithi086.stdout: "period": "43cc2be5-65c1-4813-aee6-d404c9e85e90", 2020-11-19T03:23:52.296 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:23:52.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:52.296 INFO:teuthology.orchestra.run.smithi086.stdout: "markers": [ 2020-11-19T03:23:52.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:52.297 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 0, 2020-11-19T03:23:52.297 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:52.297 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:52.297 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605756231.294848_1375.1", 2020-11-19T03:23:52.298 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:52.298 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 1, 2020-11-19T03:23:52.298 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:52.298 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T03:23:51.294848Z", 2020-11-19T03:23:52.299 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 4 2020-11-19T03:23:52.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:52.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:52.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:52.299 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 1, 2020-11-19T03:23:52.300 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:52.300 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:52.300 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "1_1605754544.294580_119.1", 2020-11-19T03:23:52.300 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:52.300 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:23:52.301 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:52.301 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "2020-11-19T02:55:44.294580Z", 2020-11-19T03:23:52.301 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:23:52.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:52.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:52.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:52.304 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 2, 2020-11-19T03:23:52.304 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:52.305 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:52.305 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:52.305 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:52.305 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:23:52.305 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:52.306 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:23:52.306 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:23:52.306 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:52.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:52.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2020-11-19T03:23:52.307 INFO:teuthology.orchestra.run.smithi086.stdout: "key": 3, 2020-11-19T03:23:52.307 INFO:teuthology.orchestra.run.smithi086.stdout: "val": { 2020-11-19T03:23:52.307 INFO:teuthology.orchestra.run.smithi086.stdout: "state": 1, 2020-11-19T03:23:52.307 INFO:teuthology.orchestra.run.smithi086.stdout: "marker": "", 2020-11-19T03:23:52.308 INFO:teuthology.orchestra.run.smithi086.stdout: "next_step_marker": "", 2020-11-19T03:23:52.308 INFO:teuthology.orchestra.run.smithi086.stdout: "total_entries": 0, 2020-11-19T03:23:52.308 INFO:teuthology.orchestra.run.smithi086.stdout: "pos": 0, 2020-11-19T03:23:52.308 INFO:teuthology.orchestra.run.smithi086.stdout: "timestamp": "0.000000", 2020-11-19T03:23:52.309 INFO:teuthology.orchestra.run.smithi086.stdout: "realm_epoch": 2 2020-11-19T03:23:52.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:52.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:52.309 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2020-11-19T03:23:52.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2020-11-19T03:23:52.310 INFO:teuthology.orchestra.run.smithi086.stdout: "full_sync": { 2020-11-19T03:23:52.310 INFO:teuthology.orchestra.run.smithi086.stdout: "total": 1, 2020-11-19T03:23:52.310 INFO:teuthology.orchestra.run.smithi086.stdout: "complete": 1 2020-11-19T03:23:52.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2020-11-19T03:23:52.311 INFO:teuthology.orchestra.run.smithi086.stdout:} 2020-11-19T03:23:52.311 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:23:52.312 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:23:52.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:52.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:23:52.314 INFO:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:23:52.358 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:23:52.358 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T14:32:40.833 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-11-19T14:32:40.872 DEBUG:teuthology.task.console_log:Killing console logger for smithi086 2020-11-19T14:32:40.873 DEBUG:teuthology.task.console_log:Killing console logger for smithi160 2020-11-19T14:32:40.873 DEBUG:teuthology.task.console_log:Killing console logger for smithi086 2020-11-19T14:32:40.874 DEBUG:teuthology.task.console_log:Killing console logger for smithi160 2020-11-19T14:32:40.874 DEBUG:teuthology.exit:Finished running handlers